Table 1. Correlations obtained for trainees in Studies II through V (a) between retention criteria

and scores obtained from the CSC test, the MIPB, end ratings by Academy instructors amd

() between MIPB scores and ratings by instructors

Trainees Terminations _Scoree
Stadt ____ —i 1

(1) II through V 214
(2) Ix through V (less Academy failures) 204
Q)- I through IV 125
(4) I (@11 trainees) 61
(5) I (trainees with CSC scores) 28
(6) IZ (Trainees with no CSC scores) 33
q@) mm 31
@) W 28
(9) V (All trainees) 89
(10) V (less Academy failures) 79
(11) V (Trainees with CSC scores) 61
(12) V_ (Trainees with no CSC scores) 28
(13) IZ+V (Trainees with CSC scores) 88
(14) If +V_ (Trainees with no CSC scores) 61

eS

HIP Instructors CSc -«Ratings va.
Scones. —MIFR__
Ek ek rp re

35 0.06 0.41 0.01
45 0.29 ~.01

33 0.10 0.38 DOL

16 0.23 0.09 0.26 0.05 0.50 0.01
7 -0,09 0:31 D.10 -0.09 0.31

9 305 0.27 D.10 0.59 0.01
7 0.39 0.05 0.53 0.05
5 9 0.50 0.01 0.07

22 0.00 0.53 0.01 0.26

2 0.02 0.28 0.05 :
4 0.08 0:47 D.01 0,22 D.11 0.36 0.05
7 -0,07 0.55 D.0L 0,47 2.05
210-033 0.40 <D.01 0,11

16 0,24 <0.08 0.39 0.01

IV. Discussion.

This followup study posed questions concern-
ing the effectiveness of predicting trainee reten-
tion in the ATC system by MTPB performance
measures, Academy instructor ratings, and CSC
aptitude test scores.

The most impressive result obtained was the
statistically demonstrated ability of the FAA
Academy instructors to predict the retention/
termination criterion, Even the nonsignificant
correlations were of substantial magnitude (0.31
and 0.27). Apparently, the instructors were able
to tap sources of variance in their association
with the trainees that were of considerable va-
lidity in estimating whether a trainee would re-
main in the ATC specialty. Probably the major
factor underlying the obtained validity was that
the potential of the trainee was foremost in the
minds of the instructors as they observed the
trainees in various phases of the Academy pro-
gram, In addition, the instructors no doubt had
opportunities to hear comments and observe be-
havior that might be reflective of attitudes and
motivations of relevance to the future perform-
ance of the trainees. Whatever may have been
the source of the validity, it was substantial, and
these results are clearly in direct support of the
earlier findings of Brokaw and Trites.*

The finding of predictive validity for the in-
structor ratings also lends support to the appro-
priateness of their use as a criterion in the earlier
study with the MTPB. Specifically, the present

results indicate that ratings of this kind are the
“best available criterion” for short term (non-
longitudinal) studies.

The interpretation of the data with respect to
the OSC scores as a predictor of the retention
criterion should give proper attention to two
factors. First, on the assumption that those
trainees who had CSC scores were selected on
the basis of the aptitude test and therefore had
suitably high scores on that test, it can be argued
that the bulk of the predictive validity of the
CSC battery would have been “used up” in the
initial screening process. Second, the trainees
in Study II had gone through additional screen-
ing both at the Academy (ie., had graduated
successfully from the Academy) and on the job
through the normal performance appraisal pro-
cess. Thus, a strong defense is available for the
argument that the retention or termination of
the Study IT trainees would be dependent on
attitudinal, motivational, and other factors that
the CSC screening battery does not purport to
measure. For these reasons, the small negative
correlation between the CSC scores and the re-
tention criterion is not surprising and should not
be interpreted as reflecting unfavorably on the
value of the CSC battery as a screening device;
the battery would have already done its job as
an initial screening instrument by eliminating
those who were least likely to succeed in ATC
work. With respect to Study V, the trainees
having CSC scores in that study had gone



and perhaps nullified because of the range re-
striction produced by the processes previously
at work in eliminating most of the trainees with
marginal aptitudes.

Il. Method.

A. Subjects.

1. Study I. This was a preliminary study
of 19 en route students, and the results were not
considered to be usable for purposes of the
followup.

2. Study II. The 60 subjects in this study
had all completed the 9-week en route training
course at the Academy and had returned to the
Academy for an interim radar course.

3. Study III. This study involved 31 sub-
jects who had the same general background as
the subjects in Study II. However, several
systematic changes in the task configuration of
the MTPB were introduced in this study so that
direct comparability of the data between Studies
II and III was not afforded.

4, Study IV. This study involved 30 train-
ees from the Academy terminal training pro-
gram. These were entry-level trainees who were
tested during the ‘first week of their 9-week
course. The study task configuration for this
group was the same as that used in Study IT.

5. Study V. This study involved an entire
class of 89 entering trainees who were tested
during the first week of the 9-week terminal
training course. The task configuration used was
intentionally different from that used in each of
the other studies.

B. Procedure. Using our roster of those
trainees who had been tested in the earlier study.
we searched the January 1, 1973, FAA personnel
roster for each participant to see if he was listed
as an ATCS; identities of those trainees who
were found on the roster were verified by social
security number.

The statistical evaluation was performed by
computing point-biserial correlations between the
retention/termination criterion and specific meas-
ures from the earlier study.

The MTPB measures consisted of an equal
variance composite of all the individual task
measures. ‘The composite was constructed by
converting each individual measure to a standard
score with a mean of 5 and a variance of 1; a

higher score reflected better performance. These
derived measures were then combined by sum-
mation.

The instructor ratings were based on a simple
form in which the instructor was asked to assign
a number from 0 to 100 (70 being passing) in-
dicating his estimate of a given trainee’s poten-
tial of becoming a fully rated ATCS. For most
trainees, ratings were obtained from three dif-
ferent instructors.

The CSC ATCS screening battery score, when
available, was obtained from the official records
for each trainee as provided by the appropriate
personnel office. (Not all trainees entered the
program through the CSC test route; ie., some
were accepted on the basis of qualifying experi-
ence. For this reason, CSC scores were not
available on a number of trainees.)

III. Results.

A. MTPB Measures. Of the 18 correlations
computed between the MTPB composite meas-
ures and the retention/termination criterion
(Table 1, column 3), only one correlation was
significant at the 0.05 level of confidence. This
was the correlation (r=0.39) for subjects in
Study IT for whom CSC scores were not avail-
able, and, hence, it is assumed that these were
trainees who were accepted into the program on
the basis of qualifying experience and not their
CSC scores. The subjects in Study II for whom
CSC scores were available yielded a nonsignifi-
cant negative correlation with the MTPB com-
posite score (r—0.09). Thus, the magnitude of
the marginal correlation (r=0.28; p<0.09) for
all subjects in Study IT should presumably be
attributed to the non-CSC subgroup.

B. Instructor Ratings. All but 2 of the 14
correlations between instructor ratings and the
retention/termination criterion were significant
at the 0,05 level or better; for those 2, p<0.10.
Of the 12 significant correlations, 3 were signifi-
cant at the 0.05 level and 9 were significant at
the 0.01 level (Table 1, column 5).

C. OSC Scores. None of the three correlations
between the available CSC screening battery
scores and the retention criterion was significant.
However, one of these correlations was of mod-
erate magnitude (r=0.22; p<0.11).


through, at the time at which we tested them,
only the screening associated with the selection
process. Thus, the small positive correlation
(r=0.22; p<0.11) is a somewhat less ambigu-
ous—though still incomplete—index of the value
of the CSC scores as a predictor of retention on
the job.

The interpretation of the results with respect
to the MTPB measures must take into account
not only the screening factors that were outlined
in connection with the CSC scores but. also the
characteristics of the studies in which the MTPB
data were collected. As noted in the introduc-
tion, this was a series of essentially exploratory
studies; in general, the question being attacked
was, Does the MTPB approach to ATCS selec-
tion offer sufficient promise to warrant further
research? Thus, as is common in exploratory
research, changes in the task configurations were
made from study to study. The reasons for the
changes were discussed in the earlier report,?
and, therefore, they will not be repeated here.
However, it is: relevant to note that the changes
were in the general nature of increasing the
overall difficulty of the task complex in Studies
TI, IV, and V. Although there is no good
rationale as to why the increased difficulty per se
would be expected to have resulted in poorer
predictive validity of the MTPB measures for
the retention criterion, it is quite possible that a
byproduct of the changes would have had an
effect. The reasoning is as follows: In all but
Study I, the trainees were fully informed as to
the intent of the study; wz, to evaluate the
MTPB as an ATCS selection device. In any
such study, it is very important that the test
device and test situation be, at least in a vague
sense, “believable” to the trainees. In retrospect,
although we did not fully appreciate it at the

time the studies were being carried out, there is
reason to suspect the difficulty of the test was
pushed to, and perhaps beyond, the believability
limit. Unfortunately, no record was made of the
best source of evidence on this point—the com-
ments of the trainees during and after the test-
ing. But, to the extent that the subjects did
react to the test situation as being unrealistically
difficult, unknown but clearly undesirable effects
on motivation would be expected.

At this time, we can marshal no factual evi-
dence to support the hypothesis that the lower
level of difficulty of the task complex in Study II
was the reason for the significant correlation be-
tween the MTPB measures and the retention
criterion. However, on purely rational grounds,
the hypothesis is not unreasonable.

With respect to the conclusions to be drawn
in regard to the MTPB measures as a predictor
of retention of trainees as ATCS’s, this followup
study inherited the features introduced by the
exploratory nature of the original study. The
most significant of these features was that
changes’ were made in the task complex from
study. to study. Thus, direct support for arguing
that the MTPB has predictive power with re-
spect to the potential of ATCS trainees derives
from the results with the subjects of Study II.
However, indirect support is also provided in
that (1) the instructor ratings served as a good
predictor of the retention criterion and (2) the
MTPB was a good predictor of the instructor
ratings in the earlier study. Thus, although it
must be concluded that final answers to the main
question posed in the study can be obtained only
through further validating research, these results
and those of the earlier studies provide a basis
for optimism with respect to the expected out-
come of such research.

REFERENCES

1, Brokaw, L. D.: School and Job Validation of Selec-
tion Measures for Air ‘Traffic Control ‘Training.
USAF: WADC-TN-59-39 (AD 214-884), 1959,

. Chiles, W. D., A. KE. Jennings, and Georgetta West:
Multiple Task Performance as a Predictor of the
Potential of Air Traflic Controller Trainees, FAA

we

Office of Aviation
1972.
3. Trites, D. K.: Problems in Air Traffic Management:
I. Longitudinal Prediction of Effectiveness of Air
Traflic Controllers, Civil Aeromedical Research In-
stitute Report 61-1, Oklahoma City, Oklahoma, 1961,

Medicine Report No, AM-72-5,



Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.
FAA-AM-74- 10

4, Title and Subtitle 5. Report Date
MULTIPLE TASK PERFORMANCE AS A PREDICTOR OF THE November 1974
POTENTIAL OF AIR TRAFFIC CONTROLLER TRAINEES : 6. Performing Organization Code

A FOLLOWUP STUDY

[8. Performing Organizotion Report No.

7. Author's)
W. Dean Chiles, Ph. D., and Georgetta West, M.S.

9. Performing Orgonization Name ond Address 10. Work Unit No. (TRAIS)
FAA Civil Aeromedical Institute
P.O. Box 25082 TT. Contract or Grant No.

Oklahoma City, Oklahoma 73125

13. Type of Report and Period Covered

12. Sponsoring Agency Name and Address
Office of Aviation Medicine
Federal Aviation Administration
800 Independence Avenue, SW. 14. Sponsoring Agency Code
Washington, D.C. 20591

OAM Report

15. Supplementary Notes

This research was conducted under Tasks AM-A-71-PSY-29 and AM-C-75-PSY-40.

16. Abstract

The current professional status of 229 air traffic controller trainees tested

in an earlier series of complex performance studies was determined by reference

to the official FAA personnel roster some 2 to 2 1/2 years later. Point-biserial
correlations between the previously obtained performance measures and the
retention/termination status of the trainees were computed; correlations were also
computed between the criterion used in the earlier study (instructor ratings of
trainee potential) and the retention criterion and between the Civil Service
Commission (CSC) air traffic control specialist (ATCS) aptitude screening battery
scores (where available) and the retention criterion. For only one group of
trainees was the correlation between the retention criterion and the performance
measure significant; this was for a subset of subjects who had not been selected
on the basis of their CSC battery scores. Twelve of the fourteen correlations
between the instructor ratings and the retention criterion were significant at the
0.05 level of confidence or better, but none of the correlations between the CSC
test scores and the retention criterion was significant. A final determination of
the potential of the complex-performance~battery approach to ATCS selection will
require further validating research. Should FAA selection policies for ATCS's
require an unequivocal assessment of the predictive efficiency of performance
tests, the results of this study and those of the earlier studies provide a basis
for optimism regarding the expected outcome of such research.

17. Key Words 18. Distribution Stotement

Document is available to the public through
the National Technical Information Service,
Springfield, Virginia 22151

Selection
Complex Performance

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No, of Pages | 22, Price

Unclassified Unclassified 4 $3.00

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized







MULTIPLE TASK PERFORMANCE AS A PREDICTOR OF THE
POTENTIAL OF AIR TRAFFIC CONTROLLER TRAINEES:
A FOLLOWUP STUDY

I. Introduction.

During the period from May 1970 to January
1971, exploratory studies were carried out to
examine the potential usefulness of a perform-
ance measurement device as a predictor of the
future performance of air traffic control special-
ists (ATCS). Five groups comprising a total
of 229 air traffic controller trainees were tested
on the CAMI Multiple Task Performance Bat-
tery (MTPB), and the predictor scores that were
derived from the performance battery were cor-
related against a criterion provided by FAA
Academy instructors, (The MTPB requires the
simultaneous performance of various combina-
tions of six tasks: warning lights monitoring,
meter monitoring, mental arithmetic, pattern
discrimination, group problem solving, and two-
dimensional compensatory tracking.) The major
conclusion drawn from that-study was “.... that
the MTPB-approach to selection offers promise
as a screening device for air traflic control spe-
cialist applicants, but further research is required
to establish this as a fact. . . This report
concerns a followup study of the trainees tested
in that program to examine the extent to which
the various previously obtained measures were
predictors of whether a trainee became and con-
tinued to work as an ATCS.

The official FAA personnel roster (dated
January 1, 1973) was screened to determine
whether each trainee who participated in the
earlier study was still listed as having the job
code of an ATCS. Thus, the elapsed time from
initial testing to the followup ranged from 2 to
21% years. Since about 90 of the trainees were
tested well along in their training program, their
tenures in the ATCS program ranged up to more
than 3 years on the followup date.

Three main questions are of interest in this
report: (1) How well do the MTPB measures
obtained in the earlier study predict the retention

of trainees as controllers? (2) How well does
the criterion (instructors’ ratings) used in the
earlier study predict retention? (3) How well
do scores on the Civil Service Commission
(CSC) ATCS selection battery (Examination
for Air Traffic Control Specialists) predict re-
tention of those trainees for whom scores were
available? :

The criterion referred to in the second question
above was the measure used in the earlier study
to evaluate the MTPB as an ATCS selection
device; the criterion consisted of ratings given
by FAA Academy instructors as to their esti-
mates of each trainee’s potential to become a
fully rated ATCS. These ratings are very easy
to obtain, and they appear to possess satisfactory
reliability (estimated to be 0.84).* As noted in
the earlier report, previous research? had shown
that instructor ratings of this general sort are
of substantial predictive value with regard to
the on-the-job performance of ATCS’s. Further
corroboration of those findings would be clearly
desirable.

It is most important to note that, in general,
these trainees represented a very select popula-
tion. At the time of testing, two of the five
groups (Studies II and III) had successfully
completed the FAA Academy training program,
which, at that time, served as a secondary screen-
ing mechanism. In addition, 28 of the 61 sub-
jects in Study II and 60 of the 89 subjects in
Study V had been selected originally on the
basis of having achieved satisfactory scores on
the CSC sereening battery (the number of train-
ees who entered the program on the basis of
CSC scores was not determined for subjects in
Studies III and IV). Thus, it would be expected
that the predictive validity of all of the various
measures used would be substantially attenuated