TABLE 2

Bescription of the 1957 and 1961 Criterion Measures

Criterion Variable

N

Moen

1957—Average Lecture Grade:
The lecture grade attained by each student in the
training course.

" 1961—WWith the FAA vs, Not with the FAA:

Individuals were dichotomized ' ax being either with
. the Federal Aviation Agency or not with the FAA.
1961—Mean Hours of Sick Leave:

‘The mean of hours of sick leave taken jn the
years 1057 Mhough 1960 was computed for each

individual.
1961—No Symptoms vs, Symptoms:

A dichotomy ting‘ medical 1 OF indi-
Viduals az Foner to wel Teponed Be tee ay
Chiefs.

1961—No Disciplinary Action vs.
Disciplinary Action:

A dichotomy representing the presence of of absence
of disciplinary actions taken as a result of conte!
errors, “Reported by facility Chiefs. for inulividuals,

\ -3-

195

188

170

149

16

16

142

M46

148

92.4

9.93

271

33.9

411

824

5.42

5.95

uO


TABLE 6

First Order and Partial Correlations Between the 1961 Criterion Measures
and Psychological Test and Biographical Measures Where the First Order
Correlations with the 1961 Criteria are Statistically Significant

Criterion +
Veriable Criterion Age toith Age

Criterion No. Description wen Ne N PortialedOut |

23 Age -233"° 49
9 Symbolic Reasoning &

Perceptual Speed (Test
| 5, Cal Capecty Que 21°° 149 -17° 135 18°
10

Abstract Reasoning, DAT 18°* 149 -96°* 135 13
" 11 Space Relations, DAT 18° 148 -08 135 17°

13 Spatial Orientation,
g AFCQT 23°° 143 -10 «135 ale

28 Previous Flying Exper. -19°* 139 67°** 195 ~05
29 Any Air Traffic Exper. 18° 139 -57°** 195 08
23
21

Age -15""
Family Relations, CTB
Cal. Test of Personality 21°* 150 -93 175 gee

SzZ 23 Age 0 100

$83 2% Education’’ 16" 168 44°* 196 ~16°
hl 23 Abe 18" 48
eS

* Polnt-blerial correlations :


TABLE 3

Most Efficient Combinations of Psychological Tests, Biographical Variables, and the 1957
School Criterion Measures for Prediction of Average Lecture Grade (Equations 1 & 4),
ite Instructc. Rating (Equations 2 & 5), and the 1957 Average Supervisor Rating

(Equations 3 & 6)

Regression Equations

Paych. Tests & Blog.

Tests,

&
1957 School

Paych.
and

Variable
No. Description

1 2 3
Bort 6B Bor

4 s_ 6
Bor Br Bor

Test Variables
1 Dial & Table Reading (AF) 20 40°° 27 37°° 22 33°°
2 Verbal Knowledge & Rea- 23 30°
soning (Test 6, Cal. Capa-
city Quest.)

7 Air Traffic Problems I

8 Arithmetic Reasoning,
AC 2A

© Symbolic Reasoning & Pex- 21 35°° 20 27°°
ceptual Speed (Test 5, Cal.
Capacity’Quest.)

10 Abstract Reasoning, DAT

17 25°° 25 3°
24 30°°

14 40°* 95 37**
98 30°°

16 25°° 24 32°
21 39°°

19 35°° 19 27°°

23 a7°°
-17 07

60 s6**

Multiple Correlations Sores S0r* a:

0 yd

“Intercorrejaiion matrix for equations was based on a common N of 135 trainees who completed the training

course. decimal points have been omitted from table entries.
4 Regression Coefficients (Betas) for standardized scores,
$ Validity Coefficients,
© Significant at less than the .05 level.
** Significant at less than the O1 level.
“Point hiserial correlation


"Statistical Methodology. — Because the re-

Port by Brokaw did not contain data suffi.
Ciently detailed to permit synthesizing proper
raw-score regression equation weights and be-

cause of a minor, but potenti important,
limitation in his analyses’ new oe, je

each of 1956-1957 criterion variables de-
scribed in Table 2. For each criterion, one
ion was based on

Grade und the Compusite
predictors, of the 1957-Average Supervisor
Rating.

The correlation matrix for computation of all

regression equations was based on a common
N of 135 subjects who had data for all variables
and had successfully completed the xraining

Making several assumptions ig the

were computed for the psychological test vari-
ables which he recommended as predictors.
This has been named Regression Equation

1

Number 7. Seven

all variables described in Tables 1 and 2 a
the seven predicted values were intercorrelat,
by a computational routine Permitting ea
correlation to be d ined on the maximu
possible number of cases having the appropria
data.

concerns, the relationships between the seven
” predicted

values and the six 1961 criterion
measures,

,


Md

TABLE 5

First Order and Partial Correlations Between the 1961 Criterion Measures,
the 1957 Criterion Measures, and Scores Predicted from Regression
Equations Where the First Order Correlations with the 1961 Criteria

ere Significant
; Criterion +
Criterion Age with Age
Criterion Variable of N +  N Partialed Out
Age -23°° 149

Average Lecture Grade °° 148 -05 135 93e°
I Composite Instructor Rating 45° 149 -18° 135 43¢°
u 1957 Average Supervisor Rating 33°° 143 ~07 135 39°°
$2” Regression Equation 5 7” 18 -4 13 14

Regression Equation 6 44° 127 -00 133 43°° .

Regression Equation 7 23°° 143 -1l 133 21°

Age -15" 169

§ Composite Instructor Raing 24° 169 -24 188 21°°
1957 Average Supervisor Rating 24°** 162 -11 170 93°°
Age -0s (169
Composite Instructor Rating 16° 169 -24 188 16°

, 1957 Average Supervisor Rating 20° 162 -N 170 20°

t ‘ .

(aga ee 168 - “
ay Average Lecture Grade 26°** 168 -14 195 24

3
Fy

With FAA
With FAA

3a.§ 4ge 02133
; 5
. iy i! 1957 Average Supervisor Rating eee 142 135 2g

& 8

Regression Equation 2 17 138 133 17°



»

PROBLEMS IN
AIR TRAFFIC MANAGEMENT:

I. LONGITUDINAL PREDICTION
OF EFFECTIVENESS OF
AIR TRAFFIC CONTROLLERS

FEDERAL AVIATION AGENCY

AERONAUT.CAL CENTER
OKLAHOMA CITY, OKLAHOMA

DECEMBER 1961 ~~ ~~


e
a i

1 ER

Fon
=

ipervisors’ Rating



ures, indicates tests related to these aptitude
areas should be included in any predictor test
bat

The only other individual psychological test
found to be significantly related to a 1961 cri-
terion was the Family Relations Scale with
Active vs Inactive Controller status. This re-
lationship was independent of age. Althou,
this suggests that personality scales may beat
some potential use, further refinement of these
measures will be necessary before they can be
recommended for operational use in a selection
test battery.

From the negative relationship between age
and the 1961 Average Supervisor Rating, it
may be inferred that older trainees are not as
likely as their younger classmates to be con-
sidered by supervisors some years later as being
among better controllers. Age, also, was

ively related to Active vs. Inactive Con-
status which corroborates the inference
that older trainees do not tend to be as success-

ful as younger men.

It was suprising. however, to find age nega-
tively related to Mean Hours of Sick Leave.
This implies ‘that older men are less often ill
than younger men. Obviously, this
further exploration of the factors behind the
relationship,

Two biographical variables, Previous Flying
Experience and Any Air Traffic
were significantly related to the 1961 Average
Supervisor Rating; but when the effect of a
was removed, these relationships
gible. Thus, none of the biographical variabl

ting previous experience was related to
sabe

went job performarce. It may be that
these perience Variables which, then and now,

are being used as partial requirements for entry
into ATC training may keep some potentially
unsatisfactory individuals out of ATC work;
but they certainly have little relationship to
subsequent ance as a controller,

Finally, Education was found to be nega-
tively related to retention in the FAA,
reasons for this are not at all clear and further
research is needed to clarify the causative
factors,

DISCUSSION

resent investigation has indicated that
ical tests can make a useful contri«

The
Psyc

bution to personnel selection for air traffic con-
trol work. In addition, it has demonstrated
that the most effective prediction of current
supervisory opinion of a controller's proficiency
can be cbtained from the controller's ATC
school instructors, assuming that the same
instructor-curriculum-student relationships have
continued to exist as prevailed in 1956-57.

The relative effectiveness of psychological
tests and the instructor's evaluation of a student
as predictors of job performance ratings are
presented in Figures 1, 2, 3, and 4. The first
two fi res were derived by identifying those
individuals in the low 25 percent of the scores
predicted from Regression Equation 7 using
psychol tests only, in the case of Figure 1,
and the low 25 per cent of the Instructor Rat-
ings made in 1956 in the case of Figure.2. The

cent of the individuals in quarter of
1961 A (Mean) Supervisor Ratings
was then determined and plttal

The second two figures were derived by
identifying those individuals who were either
in the lowest 25 per cent of the 1961 Average
Su Ratings or who had been cited for
violations of air traffic rules ‘and regulations.
Ir eee cant of these SA marginal individuals
in each quarter of the scores Regression
Equation 7 was determined and plotted in
Figure 3; similarly, the per cent of m
individuals in each quarter of the 1956 In-
structor Ratings has been plotted in Figure 4.

Considering the correlations dis-
cussed, the superior predictive efficiency of
the Composite Instructor Rating would be ex-
pected. But, it must be Temembered, these
‘predictions were made after trainees had heen
recruited, hired by the organization, and put
through a fengthy and expensive training pro-
Bram. Some loss in efficiency of prediction by
using psychological tests instead of instructors
could well be tolerated on economic grounds,

/ REFERENCES
1. Brokaw, 1...'D. School and job validation
measures ‘for aie traffle contml trainin,

30.30, Pere, wy, WADD,

AFB, Texas, 1959,

2. Greenberger, M. H. and Ward, J. H. Je An iterative
technique for multiple correlation analysis, IBM Tech.
Newsletter No, 12, IBM Co, New York, N. ¥., 2058,

3 Bottenberg, R.A. An iterative multiple: Tegression
procure wing the IBM 650. WCLLcTN.O0-. Pers
.aboratery, WADD, USAF, Lackland AFB, Tesas, 1980,

selection
/ADC-TN-
SAF, Lackland


Regression Coefficients for Standardized Scores (Betas) Reported by
Brokaw and Used to Derive Equation Number 7°

Variable

No. Description _____Regression Coefficients}
7 Air Traffic Problems I 20
8 Arithinetic Reasoning, AC 2A i
9 bolic Reasoning and Speed
(ree 5, Cal. Capeeity Quest) 16
19 Code Translation, Survey of Working S;
and Accuracy peed G

+ Decimal points conitted.
Table 5 contains the correlations between ion leaves no doubt that a valuable con-
those re, uations which were signifi- tribution ‘can be ade oe dat yatusble con-
cantly related to 1961 criterion measures traffic controllers by the use of psychological
the effect of Onl; No 8 we ‘eats

age. Only was A of the correlations between
significantly related to the elias the 1956-1957 criterion measures and the 1961

1957 criterion measures and the 1961 criteria,

also contained in Table 5, it is not surprising ite I Rating and the 1957 Av-
that Regression No. 6, which ingluded ths Composite Instructor A
ite Instructor Hating as 4 ge Sup Rating, rela

variable, had the highest correlation Erith the by the petal conga, as indicated
1961 Average Supervisor Rating, What was 5, when the influence of age was removed

surprising, was the remarkabl high correlation
between the te Instrnctor Rating and statistically
the 1961 Average Supervisor Rating.’ Obvi- Individual psychological test and biographi-
ously, the instructors ih 1956 were’ making cal measures which were significantly related
exceptionally valid judgments’ con: ming a to dny of the 1961 criteria are presented in
trainee’s potential for air traffic control work. Table 6, Two tests of the abstract reasoning
. . ype and two of the s; relations or orienta-
The correlation between Regression No. 5, te type were predictive of the 1961 Average
_ Rew! 7 eet eon Bye tests” Supervisor Rating. When the effect of age was
as predictors, and the 196! erage Supervisor eliminated from the correlation, one of the

Rating becam insignificant when age was Pat abstract reasoning tests no longer was signifi:
tiated out. However, Regression No. 7, which cantly related to the criterion. SHlowever the
utilized tests originally recommended by present findings, and the fact that Brokaw
Brokaw (1), remained significant’ when age Found these four tests significantly related to
was eliminated. The significance of the latter One or more of the ATC school criterion meas-

-6-



TABLE 1
Listing of Predictor ' wiables'

Numerical and Reasoning Tests
1. Dial & Table Reading (AF)
Test 6, Cal. Capacity
Number Series, Cal. Test of Mental
Maturity
Numerical Quantity, Cal. Test of Mental
Mai
Arithmetic, Pers. Selection and Classifi-
cation Test
6. Numerical Quantity, DAT
‘7. Air Traffic Problems I
8. Arithmetic Reasoning, AC 2A

Abstract Reasoning & Perceptual Tests
9. Test 5, Cal. Capacity Quest.

10. Abstract ing, DAT

11. Space Relations, DAT

12. Aerial Landmarks, AFOQT

13, Spatial Orientation, AFOQT

14. Instrument Comprehension AFOQT

Verbal Tests

15. Test 7, Cal. Quest.

16. Reading, Pers. Selection and Classifica-

tion Test ,
17. ge Usage, Sentences, DAT
18, Von eee AC 2A

2B wp

4

Clerical Speed and Accuracy Tests
Jv. Code Translation, Survey of Working
Speed & Accuracy
-20. Counting, Survey of Working Speed &
Accuracy

Temperament Tests

21. Family Relations, CTB, Cal. Test of
Personality

22. Nervous Manifestations, CTB, Cal. Test
of Personality

Biographical and Experience Variables
23. Age

24, Education

Marital Status

Previous Flying

Airport Traffic Control

Ground Control Approach Experience
Any Air Traffic Experience

Senior FAA Rating

I. FAA Certification in any status

SBENER

oe

t i
“Complete deveiption and sour of all poychalogcal tat varahlesevalale upon request

the men were located. Fof each of the original
men at his facility the Chief was asked to sup-
ply promotional and job information, an jindica-
tion of any disciplinary actions taken ag a result
of violations of air traffic tules or 7
and medical history information. In addition,
each Chief was asked to have four cf his
Supervisors rate each subject on aj ,

evaluation form. The form contained

~ items related to job performance, ability as a

cortroller, judigment, and Personality char-
acteristics.

From these two sources of information, six
criterion measures were developed. These

Were: (1) Average Supervisor Rating; (2)
Active vs. Inactive Controller; (3) With the
FAA vs. Not With the FAA; (4) Mean Hours
of Sick. Leave; (5) No Symptoms vs. Symp-
toms; and (6) No Disciplinary Action vs. Dis-
Action. More complete descriptions

the criterion variables are given in Table 2,
together with their means and standard devia-
tions for the maximum number-of subjects on
whom the criterion variable had been obtained,

Table 2 also contains similar information for
the criterion variables collected in 1956 and
1957, and used in the original study.

-2-


PROBLEMS IN AIR TRAFFIC MANAGEMENT;

some
ae prodicetle by da pyc oa? lafamaeton of Mel ay
In 1956, representatives of the Aeronautical PROCEDURE
Center of the Ciyil Aeronautics Administration Predictor Variables. — The Personnel Labora-

or other infor-

" USAF,
cal Center), srith members of the {r7. USAF. supplied
Fervomel Laborsery of he Users paso menses deserbed by Brokaw (1). A listing
aire management pemenet i Sample. sional 1 Offices the Federal
air traffic A re- . Re; F
selectoe viet was asranged and Ll Aviation Agency (FAA) were able to s
selection testing of air con (ATC) current FAA facili
nautical Contetis Cae eee atthe Acro mation, on all but 10 of the original 197 sub-
nautical Center in Oklahoina City. results jects. Since no information was available for
of this project have been reported by Brokaw these 10 mén, their data were excluded from
(1). His findings indicated that all regression analyses. OF the remaining 187
grades and instructor ratings, obtained im- subjects, 16 had failed the training course and

The present investigation represents a five
year follow-up of the subjects tested in 1956
“to determine if current job performance evalua-
tions, retentiontin ATC work, incidents of un-
satisfactory ATC work, and medical his
information could be predicted by the data coi-
‘ected in 1956 and 1957.

‘

left the FAA early in 1957, 15 who had

the training course had left the FAA, 2 were
deceased, replies were not received for 2, and
3 were with the FAA but no other information
was available, This left 149 subjects (includ
ing 4 training’ course failures still with the
FAA) for whom relatively complete criterion
data were obtained.

Criterion Data Collection Method. — A letter
describing the project and data collection forms
was sent to the Chiefs of the facilities at which

-1l-


r

ba
|

|

J

j

 otetimiatemneeememnenl

gern:

ro we

PROBLEMS IN AIR TRAFFIC MANAGEMENT:
IL LONGITUDINAL PREDICTION OF
EFFECTIVENESS OF AIR TRAFFIC CONTROLLERS

a :

DAVID K. TRITE:,7uD.*

611

FEDERAL AVIATION AGENCY
AERONAUTICAL CENTER
OKLAHOMA CITY, OKLAHOMA

December 196)


SCORE
RANGE

“AC 61-800

FIGURE 3 .
satisfactory controllers tn approximate fourthe,

of scores predicted from regression equation. ,
SATISFACTORY : MARGINAL N
, 78% 36
63% 35
58% 36
50% 36
FIGURE 4
* Percent A cartel ond satisfactory controllers in epproximete fourths
SATISFACTORY MARGINAL
CMLL N
\
. 33
- $
40
56% 34
40% 38