Ficure 1, The performance panel.

One of the four remaining tasks also involved
monitoring but, in this case, of a static process,
This task, called warning lights monitoring, con-
sisted of two aspects. The first aspect was rep-
resented by five red lights—one at each corner
and one in the middle of the panel. The subject
was instructed to push the button below a given
red light any time that light was illuminated.
The second aspect consisted of green lights paired
with the above mentioned red lights. With this
task, the subject was required to push a button
below a given green light to re-illuminate that
light if it should go off. For each of these tasks,
response times were measured in tenths of a
second from the onset (or offset) of a light until
the subject returned the light to its normal con-
dition by pushing the appropriate button.

The third task, mental arithmetic, required the
subject to sum two three-digit numbers and sub-
tract a third three-digit number from that sum.
He entered his answer by appropriate manipula-
tion of a set of three decade pushbuttons for the
first three digits and a three-position, center-off
switch to indicate whether the fourth digit was
a “0” ora “1,

Problems were presented at a rate of three per
minute; performance was scored in terms of the
percentage of problems correct.

The fourth task was called target identification.
This task required the subject to view a standard
“target image” and then decide whether the first,
second, or neither of two comparison images was
the same as the standard image. The task was
made somewhat more difficult by random distor-



conditions; performance under this condition
was intermediate during both 2-hour periods of
the “without-code-lock” performance and was
superior to the other two probability rate condi-
tions during the first 2 hours of the “with-code-
lock” performance. When the “with-code-lock”
data were broken down by crews a puzzling re-
sult was obtained. In no case was the prob-
ability signal-rate effect. significant with respect
to target identification performance for the Able
crew. For the Baker crew the effect was sig-
nificant for the first 2 hours (F = 6.81;
d.f. = 1,6; p is less than .05) with best perform-
ance associated with the slow probability rate
condition. During the second 2 hours, the effect
was also significant (Chi square r= 8.4;
p = .0085), but best performance in this case was
associated with the fast probability rate condi-
tion.

The task complee. The capriciousness with
which the probability monitoring signal-rate
variable appeared to affect. performance on the
battery as a whole is strongly suggestive of some
uncontrolled group factor combined with the
primary experimental conditions. There is a
“built-in” possible provision for such a factor in
the structuring of the groups as “crews” with a

“crew commander.” The two tasks on which
the commander might be expected to exert in-
fluence are target identification and code-lock.
In the case of target identification, the auxiliary
display used by the commander in making his
final crew response on this task provided him
with complete information as to how each sub-
ject was doing on a problem by problem basis.
Previous research has shown that the commander
can and does develop expectations with respect
to the capabilities of the individual subjects in
performing this task. All that remains is for
the commander to provide feedback to the indi-
vidual subjects with some sort of commentary
as to whether the subject is or is not doing as
well as the commander thinks he should be doing.
In this regard, the commander of Baker crew
commented on the performance of the other crew
members much more frequently than did the
commander of Able crew.

The manner in which code-lock performance
might be affected by the commander is straight
forward. Because of the structure of the task
and the search procedure, the commander is able
to determine via the intercom how effectively
each subject is contributing to the solution of the
code-lock problems. Again, as with target iden-

‘Taste 6.—Per Cent Correct For ARITHMETIC AND TARGET IDENTIFICATION Tasks.

Signal Rate
Task Period Chi Square r P
‘Training Slow Fast
Arithmetic. ..._.--.--.-----------------| Ist 2hrs. 90.7 91.8 91.8 20
2nd 2hrs. 90.7 90.9 90.7 65
4-hour 90.7 91.3 90.7 +20
session
Arithmetic with Code-lock_-_------------| 1st 2hrs. 81.1 83.6 86.9 4,85. <0
2nd 2hrs. 85.1 84.9 84.9 «65
4-hour 83.1 83.8 85.9 1.08
session
Target Identification... ...-.-------------| 1st 2hrs. 83.7 82.3 81.3 +45
2nd 2hrs. 82.0 83.0 85.0 «80
4 hour 82.8 82.6 83.1 3.35 <20
session >
Target Identification with Code-lock_-..--_ Ist 2hrs. 67.3 75.0 66.3 4,85 <0
2nd 2hrs. 71.3 67.7 76.0 6.65 <.05
4 hour 69.3 71.3 TL «15
session



stations one at a time for purposes of going to
the rest room or getting a drink of water during
the period beginning at 105 minutes and ending
at 180 minutes. The experiment consisted of
three sessions: first, a baseline session in which
signals were introduced on the probability moni-
toring task at the rate used during training;
second, a session in which signals were introduced
at a decreased rate; and third, a session in which
signals were introduced at a rate faster than the
one used during training. The characteristics
of the three distributions of signals are shown
in Table 2. The resultant signal rates were:
Training—15.5 signals/hour; Slow—9.4 signals/
hour; and Fast—20.6 signals/hour; the cor-
responding mean intersignal intervals were: 232
seconds, 381 seconds and 175 seconds.

Tare 2.—InrersiGnat Disraipurions ror THe THREE
Sranau Rares.

Signals / hr.
Frequency Slow ‘Medium Fast
on (15.5) (20.6)
4 260 156 104
3 286 182 130
3 338 208 156
2 390 234 182
2 442 260 208
1 494 286 234
1 546 338 260
1 598 364 286
1 650 416 312
Mean Interval 381 232 175
Range 390 260 208

Prior to the beginning of the experiment, sub-
jects were given extended training on the task
complex. The specific amounts of training re-
ceived by the two groups were as follows: four
of the subjects in Baker crew received 48 hours
of training (12 sessions) and one subject, who
missed one session, received 44 hours of training;
in Able crew three subjects received 44 hours of
training, one received 24 hours and one 16 hours.
Both of the subjects in Able crew who received
the lesser amounts of training were “good sub-
jects” and were judged to be adequately trained
by the beginning of the experiment. In this
respect, the remaining subjects were undoubtedly
over-trained.

The five subjects in Able crew were routinely
tested on Monday, Wednesday and Friday morn-
ings from 8:00 to 12:00. Those in Baker crew
were tested on Tuesday, Wednesday, and Thurs-
day afternoons from 1:00 to 5:00. These were
also the days and times at which the test sessions
in this experiment occurred.

The only information that the subjects were
given about the nature of the experiment was
that it involved the work characteristics of the
tasks. They were simply told that the training
phase had ended and the test phase was begin-
ning. At this time, the bonus procedure was also
explained to them. .

IIL. Results and Discussion.

Probability monitoring. The analysis of the
probability monitoring data had to take into
account the fact that a signal on this task re-
mained until (a) the subject detected and elimi-
nated the signal, or (b) the stimulus program
called for a new signal to be introduced. As a
result, changes in signal rate (mean intersignal
jnterval) could directly affect the proportion of
signals detected. The only assumption required
to support such a prediction is the very reason-
able one that the probability of detecting a given
signal is proportional to the length of time for
which the signal is present. A detection time
measure based on only correct responses could
be affected in an opposite manner. These pos-
sible direct effects were precluded by adopting
an after-the-fact cutoff time for purposes of the
analysis with no change in the task as viewed
by the subject. Specifically, any response time
greater than 104 seconds (the shortest intersignal
interval of the fast signal-rate condition) was
treated as a missed response in the primary
analysis.

Separate nonparametric analyses (Friedman
two-way analysis of variance by ranks) were
carried out on the detection time measure (with
the cutoff score) for each of the six conditions
under which probability monitoring was per-
formed as well as for the first 2 hours, the second
2 hours and the entire 4 hours of the sessions.
The mean values for each condition and the re-
sults of the analyses are presented in Table 3.

The only performance condition under which
the value of Chi square r reached the .05 level
of confidence was that in which probability
monitoring was performed concurrently with the



tion of the comparison images. Problems were
presented at a fixed rate of two per minute. The
presentations of the standard and two compari-
son images took place during the first 13 seconds
of the problem period, and the response could
be made any time after the second comparison
image appeared, ie, any time during the final
19 seconds of the problem period. The subject
indicated his answer by depression of the appro-
priate one of three buttons marked “1”, “2”, and
“N” (Neither). After the subject designated as
the crew commander had made his response, an
auxiliary display was illuminated at his crew
position to indicate to him the choices of each
member of the crew (including his own). Mak-
ing whatever use of this information he wished,
he then made a final selection of an answer “on
behalf of the crew.” The choice made as the
crew response by the commander was then dis-
played on a three-light display at each individual
crew position. Only the crew commander re-
ceived feedback as to which answer was correct,
and it was his option as to whether or not to
pass this information on to the remainder of the
crew. Performance on the task, both individual
and crew, was measured in terms of the number
of correct responses.

The fifth task, called code-lock, involved group
performance. This task required the subjects to
find the correct sequence in which each subject
should push the code-lock button located on each
subject’s panel; entering the correct sequence
illuminated a green light. The onset of a red
light on each subject’s panel indicated to the
subjects that a problem was present. When the

subject who was number one for that problem
pushed his button, the red light would go out
and remain out. If the next subject to push was
any other than the number two subject in the
sequence, the red light would come back on, thus
providing error feedback to the group and indi-
cating that the number one subject would again
have to push his button so that the search for
the number two subject could continue. The
process continued in this manner until each sub-
ject had pushed his button once and only once
in the correct sequence at which time the green
light was illuminated. After a 30-second delay,
the same problem was presented again and the
subjects were to enter the previously obtained
solution as rapidly as possible without error.
Then, after another 30-second delay, a new prob-
Jem was presented. The subjects used an ear-
phone intercom system to coordinate both their
initial search for the correct sequence and the
entering of their second solution. The crew com-
mander was responsible for assuring that a co-
ordinated attack was followed in the search
procedure and in entering the second solutions.
Measures were made of the time required to
reach the first solution, the time required to enter
the second solution, the total number of errors
and the total number of responses for the first
and for the second solutions.

Procedure. Performance was measured on the
combinations of tasks shown in Table 1; each
test session lasted 4 hours and consisted of two
of these programs in succession. Although no
break was provided in the 4-hour schedule, sub-
jects could, with permission, leave their duty

Taste 1.—Basic 2-Hour Tasx-PerrorMance ScHEDULE

Minutes
‘Task ) 1 30 45 60 15 90 105 120
Monitoring. -.....-_------------------ XXX XXX XXX XXX XXX XXX XXX XXX
Atithmetic Computation. . ‘XxX REX
Code-lock Solving--.-_----------------- XK XXX XXX EX
Target Identification. ....._....--.---.- xxx xEx
Level of Demand on Performance... -.-- low med hi med med hi med low

Norm: Hach “x” in the table represents 5 minutes.


of the session, the second 2 hours and the entire
4 hours. The only instance in which response
time was significantly affected by the differences
in signal rate on probability monitoring was
again in the case of the condition in which target
identification was performed in combination with
the code-lock task. However, in contrast to the
finding for the probability detection time meas-
ure, the slow signal rate on probability was asso-
ciated with faster response times to the onset of
red lights.

In none of the previous studies has there been
any indication that good performance on either
the red lights or probability monitoring tasks
is achieved at the expense of the other. And a
rational analysis of the requirements of the two
tasks would not lead to any such prediction.
This again leads to the speculation that some-
thing unexpected happened with this particular
task combination.

Data are shown in Table 5 for the offset of
green lights. The only significant variation for
this task was found for the data combined over
the first 2 hours of the session. However, the
mean values for the combined data for the second
2 hours showed a numerical—though not sig-
nificant—reversal of the relation between the
slow probability rate condition and the other
two conditions. The data for the first 2 hours
for each of the three conditions were further
broken down in two ways—by 15-minute inter-
vals as shown in Table 1 and by whether per-
formance was or was not concurrent with code-
lock. The first 15-minute period was the only
one for which significant differences were found

TasBLe 5.—Rusponsy TIMES IN

(Chi square r = 8.55; p is less than .02). How-
ever, the mean values for offset of green lights
did not correspond to those for the 2-hour data;
the means were 2.4, 2.7, and 1.1 seconds for the
training, slow- and fast-probability rate condi-
tions, respectively. Considering the second break-
down, the combined data for the four periods
during which the code-lock task was not per-
formed showed no effect of the probability rate
variable; the mean values were 3.7, 2.1, and 4.9.
For the four periods when code-lock was per-
formed, the effect was significant (Chi square
r= 7.8; p =.02). The mean values in seconds
were 5.3, 2.4, and 2.8 for the training, slow, and
fast probability conditions. When the data for
the with-code-lock condition are further broken
down by groups, it was found that the changes in
the signal rate on the probability monitoring task
affected the Able crew significantly (Chi square
r= 7.6; p = .024) but not the Baker crew (Chi
square r = 2.8).

Mental arithmetic and target identification.
The data summarizing the performance of these
two tasks, each of them with and without con-
current code-lock performance, are shown in
Table 6 for the first and second 2-hour periods
and for the entire 4-hour session. The only sig-
nificant effect of the probability monitoring
signal-rate variable was for the target identifica-
tion task during the second 2 hours when per-
formance was with the code-lock task. Inspec-
tion of Table 6 shows that this was the only
2-hour period in which performance of target
identification under the slow probability rate
condition was inferior to the other signal rate

Szconps to Green Licuts.

‘Task

Signal Rate
Chi Square r

‘Training Slow Fast

Monitoring only-.
Arithmetic*_ .
Arithmetic and Code-lock
Code-lock
Target Identification and Code-lock
Target Identification.
ist 2 hours
2nd 2 hours.
4 hours...

ot

Hanon mooa

<20
<.20

err

<01

PSP RR eN
Seppe er
BORN NH ON oD
Se Ps aL ey
Raeeconne
eB
Bee dooney

+08

*Norw: The monitoring tasks were performed with all task combinations.

7



AM 69-6

METHODOLOGY IN THE ASSESSMENT OF COMPLEX
HUMAN PERFORMANCE: THE EFFECTS OF SIGNAL
RATE ON MONITORING A DYNAMIC PROCESS

W. Dean Chiles, Ph.D.
Cheryl Beveridge Bruni, B.S.
Robert A. Lewis, B.S.

Approved by Released by
lit Pte. PU segek mo
J. Rosert Diirz, M.D. P. V. Smerz, M.D.
Cuter, Crvin ArromeEpican Fepera, Arr Sureeon
InstrruTe
April 1969

Department of Transportation

FEDERAL AVIATION ADMINISTRATION
Office of Aviation Medicine



Taste 7.—~Two-Hour Test-Retest RELIABILITIES: PROBA-
BILITY Monrrorine Detection Times with Curorr
Score.

Training | Slow Fast
Spearman r_.----.-------- «770 «742 1924
Product-Moment r_..------ 877 527 964

Taste 8.—Two-HouR Trst-Rerest RELIABILITIES : PROBA-
BILITY Monrrorine Derection Times witHour CUTOFF
Score.

‘Training | Slow Fast
Spearman r_____---------- .879 | .855 | .967
Product-Moment r-..--.---| 868 ] .755 | .951

The first point to be noted is that the reliability
coefficients are quite respectable in all but one
case considering the fact that they are based on
an N of 10. The exception is seen in the product-
moment correlation for the slow signal rate
shown in Table 7, and this low value relative to
the other coefficients is produced by the curtail-
ment of the distribution resulting from the ap-
plication of the 104-second cutoff score, a pro-
cedure that is peculiar to making comparisons
among signal rates.

The data of primary interest are those of
Table 8 which did not involve a cutoff score;
and, of these coefficients, the product-moment
correlations are of more interest in that they can
legitimately be used in developing a regression
equation. All three of these coefficients are sig-
nificant at the .05 level or better as evaluated by
t-tests based on a hyperbolic tangent transforma-
tion of the coefficients.© Using the same trans-
formation, the difference between the reliability
coefficient for the slow-signal rate (.755) and the
fast-signal rate (.951) does not quite reach sig-
nificance. The value of t (.10) is 1.895 and
t (.05) is 2.365, each for seven degrees of free-
dom; the t value for the obtained difference was
2.254,

If it is assumed that increasing the signal rate

to 21/hour does not affect the performance of
this task other than to decrease the time required

10

to present a given number of signals, then the
data from this experiment can be compared with
the data from previous research. Specifically,
in a previous experiment using what is here
called the slow-signal rate, a correlation of .76
was obtained between successive 2-hour periods
of performance (i.e. 19 signals) based on an N
of 20 subjects. Correlations were also computed
on these same 20 subjects for 9 hours of per-
formance (85 signals) ; in this case the reliability
coefficient was .945, and the difference between
these two correlations is highly significant. Thus,
at some point in the vicinity of 40 signals, the
reliability of this task appears to approach a
maximum.

IV. Summary and Conclusions.

This experiment examined the performance of
two five-man crews of highly trained subjects on
a task requiring the monitoring of a dynamic
process. The subject was to determine whether
or not the mean position of any of four contin-
uously fluctuating pointers had deviated from
zero. This task was performed in a time-shared
manner as a part of a task complex which also
involved the monitoring of static processes, men-
tal arithmetic, visual discrimination, and group
problem solving.

The variable manipulated was the rate at which
signals were presented on the dynamic monitoring
task. Three rates were used: 9.4 signals per
hour; 15.5 signals per hour (also the rate used
during training); and 20.6 signals per hour.
Performance on the dynamic monitoring task
varied significantly over the three signal rate
conditions with only one combination of tasks—
when performance was concurrent with the visual
discrimination and the group problem solving
tasks. Poorer performance was associated with
the slowest signal rate.

One of the two subtasks involving the moni-
toring of static processes (reaction time to the
onset of a red light) showed significant variation
with the same combination of tasks; however, in
this instance, better performance was associated
with the slowest signal rate. The other subtask
(reaction time to the offset of a green light)
showed significant variation in average perform-
ance for the first 2 hours. Better performance
was associated with the slow-signal rate on the
dynamic monitoring task; but, upon further
breaking down the data, this finding held only


tification, the commander of Baker crew was
apparently much more willing to impose his
thoughts and suggestions on the remainder of
the crew than was the commander of Able crew.

Unfortunately, both crews began to place
greater emphasis on speed in solving the code-
lock problems at about the beginning of the
study. Baker crew gave evidence of this in their
within-crew conversations one or two sessions
before the end of the training phase, and Able
crew began emphasizing speed during the first
session of the study proper. Thus, it is entirely
possible that the obtained effects of the prob-
ability monitoring signal-rate variable only ap-
peared to be effects of signal rate and that the
actual processes underlying the changes in per-
formance were crew or crew-commander effects.
Alternately, it could very reasonably be argued
that finding four comparisons to be significant
at the .05 level or better out of a total of 39 com-
parisons would best be interpreted as a result of
chance. From this point of view, then, it would
be argued that the experimental design had in-
sufficient power to reveal the effects of signal
rate. Or, it could be argued that differences in
signal rate, within the range of values used in
this experiment, have little or no effect when the
monitoring task in question is performed as a
part of a task complex over a relatively long
period of performance. The only real reason to
doubt that the results were chance is the fact
that the significant effects were consistently as-
sociated with the target identification/code-lock
task combinations.

Good performance on the probability moni-
toring task is primarily dependent on the devel-
opment of efficient scanning habits in time-
sharing this task with the remaining tasks of the
battery. If the training resulted in the develop-
ment of a scan frequency that was high enough,
then neither increasing nor decreasing the signal
rate should have much of an effect on detection
times. Obviously this argument rests on some
sort of assumption to the effect that the require-
ments of the battery as a whole are sufficiently
demanding that the subjects do not develop ex-
pectancies with respect to when the next prob-
ability monitoring signal will occur—or at least
if they do develop them, the expectancies are

weak enough that the subject’s scanning habits
are not altered. The better performers on the
battery as a whole (and to some extent all sub-
jects) do in fact develop regular scan patterns
based largely on the “gaps” in the on-going
stimulus program of the task complex. For ex-
ample, after a suitable amount of training, the
average subject can complete ‘an arithmetic prob-
Jem in about 10 seconds or less. With prac-
tically all subjects, at least a portion of the
remaining 10 seconds before the next arithmetic
problem is presented is spent in scanning the
displays of the monitoring tasks and the prob-
ability meters in particular, Thus, the typical
trained subject scans the meters once and _per-
haps twice every 10 to 15 seconds while perform-
ing the arithmetic task. The scan frequency is
slightly lower during the time when the target
identification task is presented, and it is much
less regular when target identification is pre-
sented concurrently with code-lock and when
arithmetic is presented with code-lock. However,
even under the heaviest workload condition, the
scan frequency is high enough that changes in
signal rate should not affect the probability of
detecting biases on the meters and should not
have much of an effect on detection time. Un-
fortunately, these comments about scanning be-
havior are not based on quantitative data. They
are based only on observations of and discussions
with the subjects.

Stability of measures. The best estimate of
the stability of the response-time measure on the
probability monitoring task is provided by the
reliability coefficient based on the shortest per-
formance interval of interest. For the purposes
of most of the projected research with this bat-
tery of tasks, the shortest period over which
performance will be measured is 2 hours. There-
fore, reliability coefficients were computed based
on, the first 2 hours of a session versus the second
2 hours for each of the probability monitoring
signal-rate conditions. -. These coefficients are pre-
sented in Table 7. Both the Spearman r and the
product-moment r are shown for each condition.
(The mean values corresponding to the data used
in computing these coefficients are shown in
Table 3.) Table 8 presents the analogous co-
efficients based on the response-time measure
without application of a cutoff score.



visual discrimination task (target identification)
and the group problem solving task (code-lock).
The detection times for the fast-signal rate were
approximately 38% shorter than for the slow-
signal rate and 28% shorter than for the inter-
mediate-signal training rate. However, the de-
tection times for the slow and the intermediate
rates were respectively 48% and 40% longer than
the corresponding 4-hour means. In contrast to
this, the typical finding in previous studies was
a difference between 4-hour mean and target-
identification-with-code-lock mean of 15% or less
(with the longer response times being associated
with the target identification condition). This
leads to the speculation that whatever happened
with this combination of tasks is perhaps not
properly attributable to signal rate on the prob-
ability monitoring task. More will be said on
that point subsequently.

No effects were found for the percent correct
detections measure; and, therefore, only the
4-hour means will be given; they were 87.6%,
87.6% and 88.9% for the training, slow, and fast
rates of signal presentation, respectively. As has
been found in previous short-term studies with
these tasks, the majority of the subjects (in this
case 7 of 10) missed virtually no signals. The
2-hour means for each of the remaining three
subjects ranged from 59% to 92%, 62% to 73%,
and 42% to 72%. The high and low values for
these three subjects were not systematically re-
lated to the signal rate conditions.

Warning lights monitoring. The mean re-
sponse times to the onset of red lights are shown
in Table 4 for each of the three signal rates on
the probability monitoring task. Means and Chi
square r’s are shown for performance under each
of the six task combinations, for the first 2 hours

TaBLE 3.—Derection Times rn Seconps ror PropaBiLiry MonrrorInG PERFORMANCE.

Signal Rate
‘Task Chi Square r P
‘Training Slow Fast

Monitoring only. - 21.4 24.9 21.6 3.2
Arithmetic*___ - 24,5 24.9 23.4 2.6

Arithmetic and Code-lock. - 27.9 31.2 31.4 14

Code-lock.. - 24.0 22.6 22.8 4,2 <.20
Target Identification and Code-lock-_ - 35.5 40.0 24,5 TA <.05
Target Identification. - 26.6 26.5 24.6 1.4

Ast 2 hours---- 24,2 27.4 24.6 2.6
2nd 2 hours. - 26.4 26.7 23.7 2.4
4 hours___ --- 25.3 27.0 24.3 2.6

* Nora: The monitoring tasks were performed with all task combinations.
Tas 4.—Ruesponss Times in Seconps To Rep Ligurs.
Signal Rate
‘"Pask a_ Chi Square r P
‘Training slow Fast

Monitoring only~ - 9 9 1.6 3.35 <20
Arithmetic*___ - 1.6 1.2 1.0 4,65 <10
Arithmetic and Code-lock.. - 1.0 La 1.2 4,85 <10
Code-lock__----_-- - 1.0 11 1.0 +95

Target Identification and Code-lock_ - 2.2 1.0 1.4 8.15 <.02
Target Identificatior - 9 1.0 la 3.75 <.20
1st 2 hours. _ - a4 1.0 1.4 65

2nd 2 hours. - 1.0 La. 1.0 4,55 <.20
4 hours. 1.2 1.0 1.2 1.4

* Norm: The monitoring tasks were performed with all task combinations.

6


presented; and, hence, stability might become a
problem. Because of the amount of training re-
quired for mastery of the full battery of tasks,
increasing the number of subjects is not a prac-
tical way of achieving greater stability. Thus,
the third purpose of the experiment was to
examine the feasibility of increasing the signal
rate on the probability monitoring task as a
technique for assuring stable measures with rela-
tively short periods of performance.

In summary, the following questions were
being asked in the experiment:

1. What is the effect of signal rate on a task
involving the monitoring of a dynamic process
when that task is performed as a part of a task
complex?

2. What are the effects of the above variable
on the other tasks of the task complex?

8. Does increased signal rate result in in-
creased stability of the performance measures
for short (2-hour) periods of observation?

Ii. Method.

Subjects. The 10 subjects were male students
from the University of Oklahoma. They were
paid for their services at the rate of $9.00 per
4-hour session. In addition, the subjects were
told that if their performance for a given session
was maintained at or above a level that was
specified only in a general sense, they would
receive a $2.00 bonus for that session. Payment
of the bonus and knowledge as to whether or
not it had been earned were withheld until the
end of the experiment. Subjects were required
as a condition of participation to agree that they
would be paid only if they completed the entire
series of experiments of which this study was
the first; they would receive nothing if they
dropped out prior to completion. Two subjects
did drop out because of the “pressures of school
work.” These two subjects were replaced with
new subjects, also from the University of Okla-
homa.

The subjects were divided into two five-man
crews; and, shortly before the end of the training
phase, the best performing subject in each crew
was designated as the “crew commander.” (His
rather simple duties will be outlined under the
descriptions of the target identification and code-
lock tasks.)

Apparatus. The specific tasks are described
in detail in a previous report by Chiles, Alluisi

and Adams? (pps. 152-158) ; and, therefore, only
the probability monitoring task will be fully de-
scribed here. This task required the subject to
monitor four displays consisting of microam-
meters. The indications on each of the meters
were programmed independently to yield a nor-
mal distribution of pointer positions for each
meter ranging from —75 to +75 with a mean
of 0; (the maximum scale values were -+100).
Each 0.5 second the meters were given a new
instruction selected from 11 possible pointer po-
sitions with damping used so that a smooth
pointer movement resulted. The critical signal
to which the subject was to respond was a shift
in the mean of the pointer positions from a
normal mean of zero to either plus or minus 25,
a shift of one standard deviation in the basic
distribution. The shift was produced by adding
a constant positive or negative voltage so that
the range remained the same.

The subject was instructed to scan the meters
systematically and, when he felt a particular
meter to be biased, to watch that meter more
closely until he had decided that it was or was
not biased. The subject was also told that he
should take action only if he was reasonably
certain that a bias was present. He tested his
hypothesis that a bias was present by throwing
a three-position, center-off, lever-type switch in
the direction he suspected the bias to lie. The
meter would then automatically come to rest on
its “true” mean, thus giving him immediate feed-
back as to the accuracy of his decision. If there
was a bias present and the subject threw the
switch in the correct direction, the bias was
automatically removed when the subject released
the switch. If it were biased in the direction
opposite to his choice (but this very rarely hap-
pened), the subject had to move the switch in
the correct direction to remove the bias. Any
time that a subject moved the switch for any
meter in either direction, a counter on the ex-
perimenter’s console for that subject incremented
by one count; if the direction were correct for
the meter involved (e.g. thrown to the left when
a negative bias was present on that meter) a
correct counter incremented by one count. At
the instant that a bias was introduced, a counter
for each subject began incrementing at a rate
of one count per second; a correct response
stopped the counter.


METHODOLOGY IN THE ASSESSMENT OF COMPLEX
HUMAN PERFORMANCE: THE EFFECTS OF SIGNAL
RATE ON MONITORING A DYNAMIC PROCESS

I. Introduction.

A number of the parameters of monitoring
performance have been investigated extensively ;
a recent review of this literature was published
by Jerison and Picket.* However, in the typical
such study, the subject was concerned with only
one task—the monitoring task being examined.
This is rarely the situation that is obtained in
the real world of work. Thus, such data are of
limited direct usefulness in pursuing the long-
term applied goal of research on human perform-
ance, viz, the development of techniques and
data that will permit generalization to the “real
world” domain of work behavior. Chapanis*
has discussed this problem succinctly in the con-
text of the design of equipment.

A recent paper by Chiles, Alluisi and Adams*
identified and provided a rationale for the selec-
tion of a basic core of psychological functions
held to be involved in complex work situations.
Although these functions were selected in the
context of aviation operations, they apply equally
well to many ground environment situations.
The functions enumerated are: monitoring, in-
formation processing, visual discrimination, and
procedural performance. The tasks that were
devised to assess these functions have been used
in a variety of studies concerned with (a) con-
finement (in a general sense), (b) work sched-
ules, (c) sleep loss, and more recently (d)
infectious diseases as reported by Thurmond,
Alluisi and Coates.° The findings of these studies
give strong support to the proposition that sub-
jects, through careful orientation, can be led to
accept these tasks as being in the domain of
work as opposed to the domains of experimenta-
tion or play.

Although the tasks have been found to be
essentially orthogonal with respect to the be-
havioral skills required, they are not experi-
mentally independent. For example, mental

arithmetic scores are generally better when that
task is performed concurrently with only moni-
toring tasks as compared to performance with a
group problem solving task in addition to the
monitoring tasks. An analogous finding holds
for a visual discrimination task (target identi-
fication). In the case of the monitoring tasks,
performance varies significantly as a function of
the load imposed upon the man in the form of
variations in the number of concurrently per-
formed tasks.’

Essentially all of the studies conducted with
this battery of tasks since 1962 have used vir-
tually the same stimulus programs for the moni-
toring tasks; the average intersignal interval has
been the same. Thus, although information has
been obtained on the effects of workload on
monitoring performance, information is lacking
as to the way in which performance varies as a
function of the parameters of the monitoring
tasks themselves.

Thus, two of the purposes of the present ex-
periment derived from this lack of information.
The first purpose was to determine the effects
on signal detection of variations in the intersignal
interval of a task that involved the monitoring
of a dynamic process; the task was called prob-
ability monitoring. The second purpose was to
examine the interaction between the signal rate
on probability monitoring and the other con-
currently performed tasks.

The mean intersignal interval on probability
monitoring in previous studies was 400 seconds—
a signal rate of nine signals per hour. Because of
the fact the major previous studies involved at
least 8 hours of performance per day for periods
of at least 12 days, the number of observations
per subject per day (the typical unit of analysis)
was sufficient to provide satisfactory stability of
the data. In an experiment involving a 2-hour
period of observation, only 18 signals would be



Qualified requesters may obtain Aviation Medical Reports from Defense Documentation
Center. The general public may purchase from Clearinghouse for Federal Scientific and
Technical Information, U.S. Dept. of Commerce, Springfield, Va. 22151.


for the average of those task combinations in-
volving the group problem solving task.

The other task which varied significantly as a
function of the signal rate on the dynamic moni-
toring task was the visual discrimination task.
When that, task was performed concurrently with
the group problem solving task, performance was
poorer during the slow-signal rate condition, but
only for the second 2 hours of the session. This
finding is beclouded by the fact that, when the
data were further broken down by crews, the
effect was significant for only one of the crews.
During the first 2 hours, the differences were
significant for that crew with the slow-signal
rate condition yielding the best performance,
whereas during the second 2 hours the effect was
significant but with the fast-signal rate being
associated with better performance.

The tentative explanation offered to account
for the obtained pattern of results is that (1) the

null hypothesis applies to this situation with re-
spect to the effects of variations in the probabil-
ity monitoring signal rate on the performance
of that task as well as the other tasks of the
battery, and (2) the obtained significant effects
are real and should be attributed to the group
problem solving task and possibly to the group
component of the visual discrimination task.

Thus, with respect to the major purpose of the
experiment, it is concluded that it is feasible to
achieve increased stability of the performance
measures on a dynamic monitoring task by in-
creasing the signal rate on the task. This can
be done without materially affecting either the
basic properties of the task or its relation to the
remaining tasks of the battery. This increased
stability was evidenced by the substantial (though
with these data, not statistically significant) in-
crease in the 2-hour test-retest reliability coef-
ficient on the task.

REFERENCES

1. Crapants, A.: The relevance of laboratory studies to
practical situations. Ergonomics, 10: 557-577, 1967.

2. Cums, W. D., Atiurst, E. A. and Apams, O. S.:
Work schedules and performance during confinement.
Human Factors, 10 : 148-196, 1968.

8. Hatt, J. T., Passey, G. B., and Merenan, T. W.:
Performance of vigilance and monitoring tasks as a
function of workload. AMRL-~TR-65-22 (March),
Wright-Patterson AFB, Ohio, 1965.

4, Jmnison, H. J., and Prcxerr, R. M.: Vigilance: A
review and re-evaluation. Human Factors, 5: 211-
238, 1963.

5. Suirn, J. G., and Duncan, A. J.: Sampling Statistics
and Applications. New York, McGraw-Hill, 1945.

6. Tuunmonp, J. B., ALLUISt, E. A. and Coates, G. D.:
‘An extended study of the behavioral effects of respira-
tory Pasteurella Tularensis in man, ITR-68-8 (Feb-
ruary), Performance Research Laboratory, Univ. of
Louisville, Louisville, Kentucky, 1968.

i