I

IV. Summary and Conclusions. lected for training on the basis of competitive
This study concerned the development and ratings from among candidates who met excep-
experimental validation of a novel aptitude test, tionally high qualifying standards in terms of
referred to as “Directional Headings” (or operational aptitude test screening scores and/or
DHT), for the selection of Air Traffic Control evaluations of pre-FAA experience. Despite,
Specialist (ATCS) trainees. The test requires these pre-screening effects, the DHT scores cor-
the subject to rapidly interpret letters, symbols, related .41 with an overall measure of training!
and degrees in order to determine: directional performance. Moreover, over 44 per cent of the!
headings (Part I), the exact opposites of head- 115 examinees who failed the training course
ings (Part IL), and opposites of headings under scored no higher than 29 on the DHT whereas
conditions of aural distraction (Part ITI). The over 85 per cent of the graduates scored 30 or}
DHT was administered on an experimental higher. Reliability of the instrument, as deter-
basis to several hundred men as they entered mined by correlating the scores based on even-|
basic ATCS training at the FAA Academy. numbered items versus scores based on odd,
The vast majority of the subjects had been se- numbered items, was .93. |
i

Table IV.--Form-A score distributions by dichotomized age groups for subjects
who passed or failed Academy basic training. |
|

Form A Age 35 & < Age 36& > All Ages
DHT
Score Pass Fail Total Pass Fail Total Pass Fail Tota
56 4 60
50& > 4 3 7
60 7 67!
i
159 10 169
40-49 7 7 24
176 17 193
157 16 173
30-39 25 8 33
182 24 206
60 9 69
2a9& < 38 13 51
98 22 120)
Total:
Passes 432 39 471
Fails 84 31 115
P+F 516 70 586

24


Training Marginal [ | Non-marginal
Form A Failures Passes 7 Passes
Directional
Headings Sample 2A: Terminal ATCS Trainees of GS-7 Level
Test Score (D.H. score correlates .50 with A+L and .35 with P-F status) N
12 oh, 41
40 & > 21.1% 7.0 71.9% 57
30 - 39 13 19 18 50
26 0%, 38.0% 36 0%
20) 6 9
2& <
57 2%, 17.1% 25.7% 35
Sample 2B: Terminal ATCS Trainees of GS-9 Level
(D.H. score correlates .38 with AtL Grade and .34 with P-F)
24
40 & > 85.8% 28
12
30 - 39 66.7% 18
29& < 3
20.0% 30.0% 10
Terminal Trainees of Combined Samples 2A and 2B
(D.H. score correlates .47 with A+L Grade & .34 with P-F)
40 & > 65
76.5% 85
30 - 39 30
RY: 68
12
29 &< 17.8% 26.7% 45

Ficure 8. Proportion of Terminal ATCS trainees within each of three Directional-Headings-Test-Score grouping
who failed the Academy’s basic Terminal training course, “marginally passed,” or passed with grades con
prising the approximate upper three-fourths of the A+L Grade distribution of the passes only.

the groups of En Route trainees. In the upper
set of bar graphs shown in Figure 3 for the 142
GS-7 TATC trainees, it should be noted that
572 per cent (N=20) of the 35 men having
DHT Form-A scores of 29 and less failed train-
ing. Of these 35, an additional 17.1 per cent
“marginally passed” and only 9 (or 25.7 per
cent) passed as “non-marginals.” Among the
50 having test scores of 30 to 39, a total of 31
(or 64 per cent) either failed or “marginally
passed.” In contrast, only 28.1 per cent of the
57 with scores of 40 and above failed or “margi-
nally passed.” The findings depicted for Sample

14

1B (Figure 3) illustrate that the DHT Form-:
scores also possess potential for discriminatin
between the training-performance criteria of th
Terminal GS-9’s. Seventeen of the 56 TAT!
GS-9’s either failed or passed with “marginal
grades and seven of the 17 scored 29 or less o
the DHT, six scored between 30 and 39, an
only four attained scores of 40 or above.
Relatively few of the ARTCC trainees wk
were examined with DHT Form B failed trai
ing and consequently the results shown in Figw
4 are not nearly as dramatic as those previous)
discussed for Form A. Yet, among the 187 GS-


the notation “360°” each denote “North.” Simi-
larly, the letter “E,” the symbol “>” and “090°,”
either separately or in combination, denote
“East.” Other letters, symbols, and degrees cor-
respond to “South” and “West.” In Part I of
the test, the examinee must rapidly interpret
and collate the bits of presented information in
order to ascertain whether the data are of a
conflicting nature and, if not conflicting, then
determine the directional heading to which they
correspond; in Part II, he must determine the
exact opposite of the heading: and, in Part III,
he must ascertain the exact opposite of the direc-
tional heading while being subjected to aural
distraction. In each item of Form A, the bits
of information are followed by one of five ques-
tions: “North?”, “East?”, “South?”, “West?”,
or “Conflict?”. The subject is offered two re-
sponse categories, “Yes” or “No.” Approxi-
mately one-fourth of the items present “conflict-
ing” data. For example, if the letter “S” were
oresented in an item of Part I with the symbol
‘>” and/or “270°” and if the question were
‘Conflict ?”, a correct answer would be indicated
oy marking the space under “Yes.” On the other
.and, the same conflicting data presented with
‘West?” (or “North?”, “East?”, or “South?”)
vould warrant a “No” answer.

The first tentative version of the test consisted
of only one part, with the task being essentially
he same for all 60 items. Small groups of
\cademy ATC trainees were examined with the
nstrument and, although no validation analyses
vere possible at that time, considerable range
nd variance were found in the distribution of
cores. Moreover, during test sessions, some in-
ividuals in every group appeared to become
ather confused, or frustrated, when confronted
vith those items in which “conflicting” bits of
aformation were presented and also hesitant in
aarking either “Yes” or “No” to many of those
sems where the questioned direction (e.g.,
North?”) failed to conform to the heading in-
icated by the data provided. When the CSC
sattery was being validated some eight years
arlier, reactions of much the same type had been
oted for many of the trainees during their
ssessment. with CSC 135 Following Oral Di-
xctions. Inasmuch as CSC 135 proved to be
ne of the most valid components of the screen-
ig battery, we hypothesized that many of the
‘ainees who had performed poorly on the in-

strument might have done so because they had
a “low frustration tolerance level.” Although
the latter phrase and the hypothesis to which it
alludes are rather loosely formulated, they
grossly reflect the reasoning which led to the
development of Parts II and III of the DHT.

As mentioned earlier, Part II is similar to
Part I except that in each item the subject is
asked whether a specified direction represents
the direct opposite of the heading. Part III is
like Part II but the examinee is subjected to
possible distraction by the aural presentation of
the randomly arranged words, “West,” “South,”
“Hast,” “North,” and “Conflict.” The resulting
3-part instrument, in which the examinees were
offered no response categories other than “Yes”
or “No,” was designated as “CAMI Directional
Headings Test—Form A.”

An alternate version of the test, Form B, was
eventually developed; it offered five response
categories. In each item of the latter, the “bits”
of information are presented in the same manner
as in Form A but the “question element” does
not appear and the examinee merely indicates
the answer he deems correct by marking a re-
sponse space under one of the five columns
laeled, “East,” “South,” “West,” “North,” and
“Conflict.”

Form A of the DHT was administered on an
experimental basis to a total of 586 students as
they enrolled at the FAA Academy during 20
January 1970 through 18 March 1970 in either
the basic ARTCC or TATC training course.
Form B was administered on a similar basis to
199 trainees, all of whom entered the Air Route
course on 27 March 1970, The vast majority of
the 785 trainees examined with either Form A
or Form B were new hirees and none possessed
an FAA ATCS certificate. For purposes of
analysis, however, the cases were divided to es-
tablish the samples and subsamples described
below.

A. Sample 1. Form A was administered to
388 ARTCC, or En Route, students who were
designated as Sample 1. Of the 388, one entered
training at the GS-5 level and 361 at the GS-7
level; the former case was added to the 361 and
the group was thereafter referred to as subgroup
“1A-En Route-GS-7.” The remaining 26 En
Route examinees entered at the GS-9 level.
They constituted the subgroup “1B-En Route-
GS-9.”



Appenvix A. Distributions of CSC ATC Aptitude Test Battery performance scores for the 893 experimentally-
examined entrants into Academy ATC training during 1961-1963 and for 281 operationally-examined trainees

who entered during 1970.

Academy
Trng. course
Failures

19.6% _ 19.3%

Sample on which the CSC
ATC Test Battery was
originally validated.
N=893 Enroute & Terminal
Trainees of 1961-1963.

16.9%

11.1%

8.8%

5.7%

CSC Score: 129 130 150 170 190 210
&< -149 -169 -189 -209 -229

N Pass: 13 33 52 124 135 126

N Fail: 38 46 47 51 37 25

N Total: 51 79 99 175 172 151

Sample of 281 GS-7 Enroute

and Terminal Trainees of 24.2%
1970 for whom operational
CSC scores were forwarded.
1.4%
CSC Score: 129 130 150 170 190 210
&< -149 -169 -189  -209  -229
N Pass: 4 56
N Fail: 12
N Total: 4 68

12.3%

Academy
Trng. course
Passes

6.3%

ee

230 250 N
-249° & > Total
87 52 622
23 4 271

110 56 893
42.4%
32.0%

230 250 N
-249° & > Total
85 105 250

5 14 31
90 119 281

25



comprised 72 cases, all with scores of 33 or less;
80 cases with scores of 44 and above represented
the upper fourth; the remaining 129 cases, with
scores of 34 to 43, were designated as the inter-
mediate half. Each of the three subgroups were
then reviewed to determine the proportion of
cases representing “training failures,” “marginal
passes,” and “non-marginal passes.” Retaining
the DHT Score intervals established with the
group of 281, a similar analysis was accomplished
with the group of 223 subjects for whom no CSC
Test data were available. In comparing the re-
sults obtained for the two groups (see Figure 5),
it should first be noted that only 11 per cent of
the 281 subjects for whom CSC scores were
available failed to successfully complete Acad-
emy ATC training, whereas an attrition rate of
31.8 per cent occurred for the 223 for whom no
operational aptitude screening measures were
available. The latter group also performed less
well than the former on the DHT. Although
not shown, the mean DHT score was 38.2 for the
group of 281 and 35.8 for the smaller group;
the mean difference was significant at the .05
level. Over one-half of the training failures in
each group scored 33 or less on the DHT. More-
over, over three-fourths of the trainees in either
group who attained DHT scores of 44 or greater
passed Academy training with “non-marginal”
A+L Grades. Also, the majority of subjects
within both groups who scored between 34 and
43 on the DHT were “non-marginals” whereas
relatively few (ie., 40.3 per cent and 30.0 per
cent) of those with scores of 33 or less were
represented in the upper three-fourths of the
A+L Grade distribution.

An additional analysis was accomplished on
the test data of the 281 GS-7 En Route and
Terminal ATCS trainees in order to more fully
assess the potential with which the DHT might
be used in conjunction with the CSC Battery to
further enhance the aptitude screening process.
In accordance with a procedure analogous to
that just described for the DHT scores, the
distribution of CSC scores was examined to de-
termine those cases representing the approximate
upper and lower fourths of the array as well as
the intermediate half. It was determined that
73 of the 281 subjects scored 260 or higher on
the CSC Battery, that 136 scored between 230
and 259, and that the lower fourth consisted of
72 subjects with scores of 229 or less. The cases

18

within each of the three CSC score intervals were
then subdivided in terms of the three previously
established levels of DHT performance (ie.,
score intervals of “33 &<,” “34-48,” and “44
&>”) and also in terms of the three categories
of training performance. In reviewing the re-
sults (Figure 6), it should first be noted that
there is little evidence of any significant rela-
tionship between the CSC aptitude scores and
training performance; however, previous studies
have illustrated that the power of the CSC Bat-
tery is in its successful screening of applicant
groups, whereas the differential scores of those
within the qualifying range offer little potential
for discriminating between levels of training
performance. (This is an expected result: See
The American Psychologist, 27 :236-239, 1972,
for discussions of this topic in a different selec-
tion context.)

The results depicted by the upper set of bar
graphs in Figure 6 for the 73 trainees having
CSC scores of 260 and above constitute convinc-
ing evidence that the DHT can be used effec-
tively to improve the aptitude screening process.
Ten of the 73 subjects represented in the upper
fourth of the CSC score distribution scored 33
or less on the DHT and three of the ten failed
training, three passed the course with “marginal’
A+L Grades and only four were “non-margi-
nals.” Disproportionately fewer trainees withir
the DHT score ranges of “34-43” and “44 &>’
failed or marginally passed. Similar results
were obtained for the 136 students who hac
CSC scores of 230 to 259. Thirty-five of the 13¢
scored 33 or lower on the new test and only 45.’
per cent of the 35 passed with “non-marginal’
grades, whereas over three-fourths of those witl
higher DHT scores did so. Results pertainin;
to the 72 trainees with CSC scores of 229 anc
lower follow the same pattern, but in more pro
nounced degree.

Such findings imply that the DHT is provid
ing measures of certain aptitudes other tha
those encompassed by the CSC Battery. Thi
was verified by the results of a series of multipl
regression analyses. Preparatory to the latte
analyses, intercorrelations and validities of th
DHT and the various tests of the CSC Batter.
were first computed for the group of 281 GS-7’
for whom CSC scores were available. Althoug
not presented in any table, the correlations be
tween the DHT and the different tests of th


-Isse]o 210M seseo gz 243 Jo 7 ATUO esnedaq Z eTdueg A0Z par

DOA poy Vomnee—wegezeny oe roe

ndulod JOU 919M SIOTOTSFEOO TEFAMSTG-IWTOdyy

az eTdueg Jo seseo y pue gy e{dueg jo

gosed 6 ATUO 10J e[QeT}TeAe BIOM seXOds Ise, OSD Ssnedeq peuTeIqo Jou stem SIUETOTIZSOO voy 3eT 21109,

EE

I aaed ‘y-HC = 8,6-SD TeUFMIeL ‘EZ

I 33ed ‘y-Hd = 8,/-S9 TeuFMAeL ‘yz

I a3ea ‘y-HC $,6-S9 eqnozug “aT

I qaed ‘y-Hd = 8, 2-9 @3nozug ‘yT

SiTqeraen Bday pue sTdueg

98° 68 708 9S eperp *8usy qel+*peoy
s'lez 7 41099eg 38°] OLV-9S9
ie se" 66 cBe 9S e209g 33ed ‘Bay ‘y-HC
ze" 6c" £6" 6'IT 6°07 9S III 33ea ‘y-HC
ez° ce" 96° 838° OIL T'9E 9S II 34ea ‘y-Hd
ge" ee" 18° 9° = 89" 06S sCO BE OS
€L ZS «T'6L «HT Opeap *BuaL get" pecy
70°- YT" 71% T°8tz2 69 42093e¢ 3891 OLV-0S0
se* os* st’ O'OT Z°9€ Zyl e200 Jaq “BAY ‘y-HC
sz" Sy TTT 88° orl ele = ZH III 34eq ‘y-Hd
Te Sy Ott si" 19° vET 6°€€ ZHI II 3ea ‘v-Hd
82° ze" tT RH GB TLE AUT
yy 8°06 «92 aperg “Susy qei+’peoy
“ve 8'SH% 6 ~ft099eG 389L O1lV-0S9
os* YUL SE (9% aq0og 3zeg *8ay ‘y-HC
se" £6" est ILE 97% IIL 34ea ‘y-H
ov" so 8" yt OTe «9% IL yea ‘y-H
79° os" 6s° LL TOT BSE 9
£9" eS 1°88 Z9€ epezD “Bur, qeT+*Pecv
70° er L°0% 2°89% 212 42093e¢ 3891 OLV-0S0
9" ec" 62" ZIT S°LE 79E = P409g Jted “Bay ‘y-Hd
sz 9e° 0g si £6* GET T'6E 29€ IIT 24ed ‘y-Ha
Iz Te’ €@ 6" 7B" WZ O9E 29 II 3a ‘y-Hd
yt gt! we? BLT ST SH WLE U9E
yea “iry. «= Saodg -8300§ TIT “Tr “Ws te ON
ee a ‘qqeq ‘Say Jaeqd qed
eqA0IFAD oly Te30L,
SuyfuteaL, 4089
— ——— ———

+ga1098 382] OLY-9S0 pue seanseow 3se] sduTpeeH

—= SS ane

[euoT}IeITg JO SOTITPTTeA pue sUoTIeTerI00z0qUT-~"IT ATAVL

10


The second criterion variable was “Pass-Fail
Status” or-“P-F.” Students whose records bore
the notation “Withdrawn in Failing Status”
were designated as “Fails”; those who were
withdrawn for other reasons were deleted from
the study, and all others were considered as
“Passes,”

In accomplishing some of the analyses, every
subject who passed training was categorized
either as a “Marginal Pass” or “Non-Marginal
Pass.” The “Marginal Pass” category was es-
tablished separately for each sample, irrespective
of GS level. First, the “Fail” cases were ex-
tracted; next, the A-+L Grades of all non-fail
subjects in a given sample were arrayed from
low to high, and those comprising approximately
the lower one-fourth of the grade distribution
were designated as “Marginal Passes.”

F. CSC Test Scores, It was postulated that
if the DHT scores of the pre-screened groups of
trainees should validate at statistically signifi-
cant levels for prediction of the training per-
formance measures just described, there would
be little doubt that the instrument would vali-
date at higher levels with applicant groups.
Nevertheless, additional information bearing
upon the interpretation of the attenuated validi-
sies could be obtained if the CSC Test Scores
and/or evaluations of pre-FAA ATC-related
axperience were included in the scheduled anal-
yses. Such data were available for a portion of
she trainees comprising each validation sample
yut, as will be discussed, there were reasons to
suspect bias in the data.

The development and validation of the DHT
vas undertaken in conjunction with, and as part
vf, a comprehensive project which had been
nitiated in response to a request by the FAA
Director of Personnel (PN-1) for updated re-
earch on factors associated with successful per-
‘ormance in ATC training. (Other aspects and
indings of the latter will be discussed in subse-
{uent reports.) In connection with the requested
esearch, PN-1 wrote to all regions specifying
hat CAMI be provided a report on each trainee
t time of entry into the Academy; the report
vas to reflect the trainee’s overall eligibility rat-
ag and other types of information. Most im-
ortantly, if the trainee had been assessed with
he operational CSC ATC Aptitude Test Bat-
ary, the report was to include all subscores

(ie., scores on each CSC test booklet) as well as
the complete aptitude measure.

Records were ultimately forwarded to CAMI
for about nine-tenths of the 3,579 trainees in-
volved in the larger project but many of them
were incomplete. Researchers received CSC
Test Scores for less than 52 per cent of the en-
tire group although it was estimated that 70 to
75 per cent had probably been operationally
examined with the battery. Unfortunately, in-
sofar as research purposes were concerned,
scores on the aptitude screening battery were
forwarded for a much smaller proportion of the
training failures than for Academy graduates.

In the present study, the 504 GS-7 trainees of
Samples 1 and 2 who were administered Form
A of the DHT were also administered a ques-
tionnaire which included the following two
items: (1) “Have you ever been examined with
the CSC ATC Aptitude Screening Test Bat-
tery?” (2) “If so, please indicate the number
of times you took it, and for each time, also
indicate the approximate date and the city in
which examined.” Exactly 95 per cent of the
total group of 504 GS~7’s indicated that they
had been tested one or more times. (Slightly
over 97 per cent of the 102 training-course
failures within the group claimed they had
taken the battery on an operational basis.)
However, performance scores on the battery
were forwarded to CAMI for only 281 of the
504, Moreover, the 281 represented 62.2 per
cent of the 402 pass cases but only 30.4 per cent
of the 102 failures. Inasmuch as the CSC Test
Scores were available for disproportionately
fewer training-course failures than for pass
cases, it was presumed that the analyses to be
undertaken in this study would yield grossly
attenuated validity coefficients for the opera-
tionally-derived aptitude measures and that the
correlations involving the latter and age and
other variables would also be affected.

G. Pre-FAA ATC-Related Experience. The
questionnaire which CAMI administered to the
DHT examinees also contained a section which
pertained to ATC-related experience. Spe-
cifically, the instructions requested that each
trainee indicate whether he had ever held a
license, certificate, or rating as a “pilot” and/or
in “air traffic control work” and/or in the field
of “communications.” Although 170 trainees



B. Sample 2. The 198 students assessed with
Form A of the DHT as they entered the TATC
basic training course represented Sample 2.
One was a GS-5, 141 were GS-7’s, 55 were
GS-9’s, and one was GS-11. The GS-5 was
merged with the GS~7’s to establish the subgroup
“2A-Terminal-GS-7.” The remaining cases
comprised the subgroup “2B-Terminal-GS-9.”

C. Sample 3. Only 199 trainees were exam-
ined with Form B of the DHT. They were
designated as Sample 3. The subgroups were
“ZA-En Route-GS-7” and “3B-En Route-GS-
9,” with N’s of 187 and 12, respectively.

The question had arisen during the develop-
ment of the DHT as to whether past experience
in control work would serve as a major determi-
nant of test performance. Unfortunately, of-
ficial records of the experiential backgrounds
were unavailable for many of the trainees who
took the test. However, there were reasons to
believe that, by maintaining separateness of the
test data for the GS-7’s and GS-9’s, certain
phases of the analyses might yield some infor-
mation bearing upon the issue. It was presumed
that most subjects who entered with a pay grade
above GS-7 received their appointment to train-
ing on the basis of exceptional pre-FAA experi-
ence (usually in military air traffic control work)
and that, on the same basis, they were auto-
matically exempted from the aptitude screening
requirement. Knowledge of FAA recruiting
practices suggested that the vast majority of the
GS-7 DHT examinees were probably men with
moderate-to-appreciable amounts of pre-FAA
ATC-related experience who, in establishing
their eligibility for selection, were required to
pass the CSC ATC Aptitude Screening Test.
Battery. It was recognized that the experience
backgrounds of the GS-7’s and GS-9’s could not
be considered representative of applicant groups.
However, it was reasoned that 7f the analysis
revealed no significant differences between the
mean DHT scores of the GS-9’s and GS~7’s, the
issue regarding possible differential effects of
lesser amounts of experience (than possessed by
most GS~7’s) would warrant little concern be-
cause the FAA traditionally recruits a very
small proportion of its trainees from among
those applicants having relatively little or no
previous ATC-related experience. On the other
hand, if the GS-9’s performed at a significantly

higher level than the GS-7’s, it would then be
unwise to recommend that the DHT be used as
an operational screening device.

D. DHT Scoring Procedures. A formula of
“right responses minus wrong” was employed in
the scoring of Form A of the DHT because only
two response alternatives (ie. “Yes” and “No”)
were offered for each item. Form B, with a
5-choice response format, was scored in accord-
ance with the formula “rights minus one-fourth
the wrongs.” Scores involving fractions were
always rounded to the nearest whole number.

Inasmuch as differences, as well as similarities,
characterized the tasks presented in Parts I, IT.
and III, it was postulated that the parts might
prove unequal in terms of difficulty and it was
also felt that the parts might yield contrastingly
different validities. Consequently, the three
parts were scored separately. An arithmetic
mean of each subject’s part scores, rather thar
a sum of the part scores, was obtained to serv
as the global measure. The mean score is re
ferred to in the present report as either “Thi
Total Average Score” or simply as “The DH1
Score.” The latter, as well as the part scores
were analyzed to determine their respective re
lationships with the criteria. In addition, :
number of similar analyses were undertake:
regarding the differences between the summar,
DHT measure (ie., “Average Score”) and eac]
of the part scores of the subjects and difference
between the part scores themselves.

E. Criterion Variables. Academy trainin,
officials provided CAMI with an “Evaluation o
Performance Record” for each of the trainee
who participated in the experimental testin
program. The evaluation form for the ARTCt
trainees was not identical to that employed i
evaluating TATC training-course performanc
Nevertheless, each provided grades which pei
mitted derivation of a similar, correspondin
measure of performance for each subject, rv
gardless of training option. The summar
measure, referred to as the “Combined Academ:
and Laboratory Grade Average” or “A+L
represented an arithmetical mean of two sep:
rate averages; one based on all examinatic
grades relating to academic materials, instru:
tion, and the like, and the other based on pe
formance grades for simulated air-traffic-contr
work in the laboratory.


35.1% 35.2% 35.1%
33.4% 32.9%
31.7%
127 50 177
121 17.6% 45 166
y 13.7%
12.2% 12.4% 11.3%
3%
6.9% 7.0%] 25 6.9%
| 45 69 37
10 12 35
1920304050. «19”~~«20~=30~C~SDSCSS~*«<iaSSC« SSS
&< -29 +39 -49 &> & < +29 -39 -49 & > &< -29 -39 -49 &>

N 362 Mean 37.5 SD 11.2
Enroute GS-7's

N 142 Mean 36.2 SD 10.0
Terminal GS-7's

N 504 Mean 37.1 SD 10.9
Combined Samples

AMDLE 411-"VIrectlonal Headings ‘Vest te

Sample 1A Sample 2A 1A and 2A
42.3%
35.7% 35.4%
2.19 32.9%
126 .9%
11
20
18 29
27
7 14.3% 14.3% 13.4%
11.5% 11.57 12.2%
rm 6.1%
8 8 1% 10
3]3 3.6% i
2
2 5
19 20 30 40 50 19 20 30 40 50 19 20 30 40 50
& < -29 -39 -49 & > & < -29 -39 -49 &> & < -29 -39 -49 &>

N 26 Mean 35.2 SD 12.4
Enroute GS-9's
Sample 1B

N56 Mean 38.2 SD 9.9
Terminal GS-9's
Sample 2B

N 82 Mean 37.2 SD 10.8
Combined Samples
1B and 2B

‘cure 7. Comparison of ATC trainees of GS-7 and GS-9 levels in terms of performance scores on Directional

Headings Test Form A.

19



TABLE I.—Distribution of Form-A Directional Headings Test scores for 388 ARTCC
and 198 TATC Trainees who were examined upon entry into FAA Academy training.

Sample 1
Enroute or ARTCC Trainees

Sample 2
Terminal or TATC Trainees

Form-A Graduates Non-Graduates Graduates Non-Graduates
D.H. Test Cum. Cum. Cum, Cum. Cum. Cum. Cum. Cum,
Score £ £ he £ £ ke £ £ he £ £ h
56 & > 13 329 100.0 59 100.0 3 142 100.0 56 100.0
54 - 55 1 316 96.0 3 59 100.0 4 139 97.9 56 100.0
52 - 53 10 315 95.7 3 56 94.9 5 135 95.1 56 100.0
50 - 51 17° 3050 (92.7 53 89.8 7 130 91.5 1 56 100.0
48 - 49 17° 288 = 87.5 3. 53 89.8 6 123 86.6 55 98.2
46 - 47 19 271 = 82.4 2 50 84.7 12 #117 82.4 2 55 98.2
44 - 45 16 252 76.6 1 48 81.3 5 105 73.9 1 53 94.6
42 - 43 33° 2360 «(71.7 4 47 79.7 18 100 70.4 4 52 92.9
40 - 41 32 203 61.8 1 43 72.9 ll 8257.7 6 48 85.7
38 - 39 38 171 52.0 3 42 71.2 12. 71 50.0 3 42 75.0
36 - 37 27° 133) 40.4 2 39 66.1 120059 «41.5 4 39 69.6
34 - 35 27° 106 = 32.2 4 37 62.7 12 47 33.1 2 35 62.5
32 - 33 16 79 24.0 4 33 55.9 6 35 24.7 6 33 58.9
30 - 31 14 63 19.1 3 29 49.2 9 29 20.4 2 27 48.2
28 - 29 10 «49 14.9 3 26 44,1 2 26 14.1 4 25 44.6
26 - 27 9 39 11.9 23° 39.0 8 18 12.7 3 21 37.5
24 ~ 25 4 30 9.1 5 23 39.0 3. 10 7.0 7 18 32.1
22 - 23 7 26 7.9 5 18 30.5 2 7 4.9 3 11 19.6
20 - 21 2 19 5.8 2 13 22.0 5 3.5 1 8 14.3
18 - 19 2 17 5.2 3 11 18.6 1 5 3.5 3 7 12.5
16 - 17 7 15 4.6 1 8 13.6 2 4 2.8 1 4 7.1
14 - 15 8 2.4 7 11.9 1 2 1.4 1 3 5.4
13 & < 8 8 2.4 7 7 11.9 1 1 7 2 2 3.6
*

Mean D.H. Score

Graduates: 38.5 39.0

Non-Grads: 31.0 31.0
Stnd. Deviation

Graduates: 10.2 9.2

Non-Grads: 14,3 9.9

*
Each mean and S.D. is based on ungrouped test performance data



TECHNICAL REPORT STANDARO TITLE PAGE

1. Report No. 2, Government Accession No. 3. Recipient's Cotalog No.
FAA-AM~7 2-18

4, Title and Subtitle 5. Report Date
A PROPOSED NEW TEST FOR APTITUDE SCREENING OF AIR May 1972
TRAFFIC CONTROLLER APPLICANTS &. Performing Organization Code

7. Author! s) 8. Performing Organization Report No.

Bart B. Cobb, M.S.
John J. Mathews, B.A.

9. Performing Orgenization Name ond Address 10. Work Unit No.
FAA Civil Aeromedical Institute
P. 0. Box 25082 11. Contract or Grant No.

Oklahoma City, Oklahoma 73125

13. Type of Report and Period Covered

12. Sponsoring Agency Nome and Address

Office of Aviation Medicine OAM Report
Federal Aviation Administration
800 Independence Avenue, S.W. 14. Sponsoring Agency Code

Washington, D. C. 20594
15. Supplementary Notes

This research was conducted under Tasks No. AM-A-70-PSY-23, AM~B-71-PS¥-23, and
AM-B-72-PSY -23.

16. Abstract

This study concerns the development and experimental validation of a novel
aptitude test, referred to as "Directional Headings" (or DHT), for the selection
of Air Traffic Control Specialist (ATCS) trainees. The test requires the subject
to rapidly interpret letters, symbols, and degrees in order to determine:
directional headings (Part I), the exact opposites of headings (Part II), and
opposites of headings under conditions of aural distraction (Part III). The DHT
was administered on an experimental basis to several hundred men as they entered
basic ATCS training at the FAA Academy. The vast majority of the subjects had
been selected for training on the basis of competitive ratings from among
candidates who met exceptionally high qualifying standards in terms of operational
aptitude test screening scores and/or evaluations of pre-FAA experience. Despite
these pre-screening effects, the DHT scores correlated .41 with an overall
measure of training performance. Moreover, over 44 per cent of the 115 examinees
who failed the training course scored no higher than 29 on the DHT whereas over
85 per cent of the graduates scored 30 or higher. Reliability of the instrument,
as determined by correlating the scores based on even-numbered items versus
scores based on odd-numbered items, was .93.

Air Traffic Controllers Availability is unlimited, Document may

Aptitudes be released to the National Technical

Personnel Selection Information Service, Springfield, Virginia
22151, for sale to the public. ;

7, Key Words | 18, Distribution Starement
|
| i

9. Security Classit. (of *his report) 0. Security Class

Unclassified Unclassified 26 | $3.00

Tol this sage! TR No. of Pay
i

orm DOT F 1700.7 (8-69)



Failed Passed
Trng. Course Trng. Course
21.3%
20 4%
Sample 1A: 362 GS-7 Enroute ATC trainees who were
administered Form A of the D.H. Test
Intercorrelations
Mean S$.D. I IIT Avg AWL PF
DH Pt I 37.4 9.5 .64 .58 .78 .28 .14 13.8%
DH Pt If 36.6 12.4 +82 93.31.21 13.0%
DH Pt TII 39.1 13.9 +93 637.25
DH Avg Pts 37.5 11.2 338.24
A+tL Grade 88.1 5.3 63
P-F Status 8.6%
6.6%
5.5%
3.9% 3.9%
Oy
D.H. Avg. Score
Pts. I, II & III: 14 15 20 25 30 35 40 45 50 55 Total
& < -19 -24 -29 -34 -39 -44 -49 +54 & >
N Passes: 8 7 10 19 39 72, 69 42 27 12 305
N Fails: 6 4 10 5 IL 5 5 5 4 2 57
19 8% 9 3 19.8%
Sample 3A: 187 GS-7 Enroute ATC trainees who were _
administered Form B of the D.H, Test
16.0%
Intercorrelations
Mean S.D. _If II Avg AtL PF
DH Pe T 39.0 10.0 .65 .69 .79 .45  .21
DH Pt IL 37.1 11.2 81.95 639) AL
DH Pt IIT 38.8 11.5 +92 42° 15
DH Avg Pts 37.9 10.6 44 1S 8.6%
A+L Grade 87.8 5.4 57
P-F Status
5.3%,
4.3% 4.3%
0% 1%
D.H. Avg. Score J
Pts. I, IL & IIL: 14 15 20 25 30 35 40 45 50 55 Total
& < -19 -24 -29 -34 -39 -44 -49 -54 & >
N Passes: 7 1 4 9 26 33 34 32 15 3 164
N Fails: 1 L 4 1 4 4 2. 1 3.

Frcure 1. Comparison of performance data for groups of En Route GS-7 level ATCS trainees who were admin

istered the Directional Headings Test Form A or B.

12



II was lower than that on Part I, but only one
such difference—involving Subsample 2A—was
statistically significant. (The performance means
pertaining to Form A appear in Table IT and
those relating to Form B appear in Figure 1.)
The authors also hypothesized that a sizable
proportion of the trainees would be adversely
affected by the “conflicting noise” imposed dur-
ing administration of Part III. Contrary to
expectation, the Part III mean score exceeded
that of Part II for every sample and subsample
and all mean differences except those relating to
Sample 8 and Subsample 1B were statistically
significant. Moreover, a review and comparison
of each individual’s part scores revealed rela-
tively few instances of performance decrements
from Part II to Part III. However, no definite
conclusions can be made regarding either of the
above hypotheses because the results are presum-
ably confounded by learning effects.

Scores on the separate and combined parts of
each version of the DHT validated at substan-
tially higher levels for prediction of the A+L
Grades than for pass-fail status (see Table II
and the tabular data presented with Figure 1).
For the four subgroups of Form A examinees,
the correlations between the composite DHT
scores and the A+L Grades ranged from .38 to
.50 while those relating to pass-fail status ranged
from +24 to 35. The overall scores of the 187
Form B examinees correlated 44 with the
training-course grade averages and only .15
with the alternate criterion.

All such validity coefficients pertaining to the
overall scores on either form of the DHT are
statistically significant. Each is also signifi-
cantly higher than the corresponding coefficient
obtained for the CSC ATC-Aptitude-Screening
Test score. However, the validity indices pre-
sented for both the operational screening battery
and the two versions of the DHT should be re-
garded as grossly attenuated because they are
based on data pertaining to individuals who had
been selected from among those who had fully
met the FAA’s existing ATC-personnel-screen-
ing standards. Had formulae been applied to
astimate the correction for restriction-of-range
affects, the resulting coefficients would have been
arger, and in some instances of much greater
nagnitude, than those actually obtained. For
oxample, if the correlation of .13 (see Table IT)
vetween the CSC ATC Test Score and A+L

11

Grade were adjusted (relative to the ratio of
variances in the performance measures of the
subjects of the current study and the 893 ATCS
trainees who participated in the original valida-
tion of the CSC Battery), a coefficient of .25
would result. Such estimated values are not
shown in the present report. Instead, the au-
thors have followed a more conservative ap-
proach and have presented only the un-corrected
validity coefficients.

B. Correlations Between Scores on Odd- and
Even-numbered Items. Correlations of scores
based on odd-numbered items versus scores based
on even-numbered items were computed for
Form A for the 362 GS-7 En Route trainee
subjects only. They were: .92 for Part I; .94
for Part II; .93 for Part III, and .93 for the
scores representing an average of the three parts.
Similar procedures were employed with the re-
sponse data of the 187 GS-7 En Route Form B
examinees, The resulting correlations were .93,
.96, .938, and .95 for Parts I, IJ, III, and the
combined parts, respectively. Such coefficients,
however, should be regarded as inflated estimates
of reliability because the DHT is admittedly a
highly-speeded test and it is known that most
speeded tests are apt to yield relatively high
correlations between measures reflecting per-
formance on odd- and even-numbered items,
particularly when the items, like those of the
DHT, do not represent a wide range of difficulty.

C. Effectiveness of DHT Scores in Predicting
“Fail,” “Marginal Pass,” and “Non-Marginal
Pass” Cases. The effectiveness with which the
scores on Form A of the DHT might be used to
forecast three categories of performance in the
ARTCC training course is illustrated in Figure
2. Prior to undertaking the series of analyses
which produced the results depicted in Figures
2, 8, and 4 (which all follow the same format),
we inspected the frequency distributions pre-
sented in Figure 1 and decided that a DHT
Score (i.e., an average of the part scores) of 29
represented the best choice for a “cutting score”
or theoretical screening cut. Analyses were then
accomplished using score intervals of “29 &<,”
“30-39,” and “40 &>.”

Looking first at the upper set of bar graphs
in Figure 2, it may be noted that 69 of the 362
En Route GS-7 trainees scored 29 or less on the
DHT and that 25 (36.2 per cent) of the 69



replied “No” to each of the three areas, the re-
mainder checked one or more. These responses
provided the basis for assigning each case to one
of eight mutually exclusive categories, with one
of the categories reflecting certificated experi-
ence in all three fields. An analysis was then
scheduled to assess possible effects of experience
on DHT scores.

H. Limited Research on Form B of the DHT.
As pointed out earlier, Form A of the DHT was
administered to a total of 586 traineers (Samples
1 and 2) whereas only 199 trainees (Sample 3)
were examined with Form B. Academy training
schedules were rearranged soon after Form B
was developed, thereby precluding further ex-
perimental test sessions. Several different anal-
yses accomplished on the data of Form A
examinees were not undertaken with Sample 3.
Inasmuch as it was determined very early in the
study that the two forms were highly comparable
in terms of both validity and reliability, it was
assumed that if other corresponding analyses of
data for the Form A and Form B groups were
undertaken, they also would yield comparable
findings. Research efforts were therefore focused
primarily upon Form A.

Il. Results and Discussion.

After scoring the DHT answer sheets, each
examinee’s scores on Parts I, II, and III were
averaged to obtain the overall measure of per-
formance (ie. “The DHT Score”). The first
analysis, which pertained to the Form A exam-
inees only, involved a comparison of the fre-
quency distributions of the DHT Scores of the
graduates and non-graduates of the ARTCC and
TATC training courses. The distributions are
shown in Table I. As may be noted, the grad-
uates of each training course tended to attain
higher scores than the non-graduates. The 329
ARTCC graduates averaged 38.5 on the three
parts of the test whereas their 59 attrited class-
mates averaged 31.0, Similarly, the mean score
for the 142 TATC graduates was 39.0 while the
56 who failed averaged 31.0. Both differences
were statistically significant at the .01 level.
Moreover, three of the En Route graduates at-
tained perfect scores of 60 whereas the highest
score of any attrited ARTCC trainee was 55;
one subject within each subgroup scored only 01.
Three of the TATC graduates scored 56 on the

test and none scored lower than 13, whereas the
scores for the Terminal failures ranged from 50
down to 05. Other differences between the score
distributions of the passes and fails, and their
bearing upon a recommended screening stand-
ard, will be discussed in one of the later sections
of this report.

When the frequencies of the Form-A DHT
Scores of the passes and fails were merged, the
resulting distributions (which are not shown)
proved to be much alike. Although that of
Sample 1 reflected somewhat greater range and
variability of performance, both were slightly
skewed in a negative direction. Moreover, the
difference between the means of 87.4 and 36.7
(for Samples 1 and 2, respectively) was not
statistically significant. Inasmuch as 60 repre-
sented the maximum possible (average) score on
the three parts, the range and distribution of
performance measures for each group were con-
sidered quite satisfactory in that they provided
no basis for questioning the difficulty level
and/or timing of the test.

A. Intercorrelations and Empirical Validities
of DHT and OSC Test Scores, A point which
warrants emphasis is that the three parts of the
DHT present different tasks or task situations
Considering these differences, the authors ex.
pected that the intercorrelations of the par
scores would be of relatively moderate magni
tude. The Pearson product-moment correlatior
coefficients,” reflecting the interrelationships, ar
shown, along with other data, in Table II. Th:
correlations (or “r’s”) ranged from .46 to’ .7
for Part I versus Part II, .29 to .69 for I versu
III, and from .67 to .88 between the scores 01
Parts II and III. In each instance, however
the lowest of the coefficients pertained to Suk
sample 2A. Otherwise, the coefficients range
from .64 to .77 for I versus II, .58 to .69 for
versus III, and from .81 to .88 for II versus IT
The narrow range of the latter would imply tha
the test possessed a fair degree of reliabilit;
The authors re-examined all data relating t
Subsample 2A but found no computational e
rors or reasons why the intercorrelations wei
lower than obtained for any other group.

The hypothesis that examinees would gene:
ally experience greater difficulty with Part ]
than Part I proved rather tenuous. For ever
sample and subsample, the mean score on Pa


cy

10.

REFERENCES

. Air Traffic Controller Career Committee Report:

‘The Career of the Air Traffic Controller—A Course
of Action. Executive Summary of the Report of
the Air Traffic Controller Career Committee. FAA,
Washington, D.C., January 1970.

. Brokaw, L. D.: School and Job Validation of Selec-

tion Measures for Air Traffic Control Training.
WADC-TN-59-39, Personnel Laboratory, WADD,
USAF, Lackland AFB, Texas, 1959.

Cobb, B. B.: Problems in Air Traffic Management:

Il. Prediction of Success in Air Traffic Controller
School. FAA Civil Aeromedical Research Institute
Report No. 62-2, 1962.

. Cobb, B. B.: Problems in Air Traffic Management:

V. Identification and Potential of Aptitude Test
Measures for Selection of Tower Air Traffic Con-
troller Trainees. FAA Office of Aviation Medicine
Report No, AM-65-19, 1965,

. Cobb, B. B. C. D. Lay, and N. M. Bourdet: The

Relationship Between Chronological Age and Apti-
tude Test Measures of Advanced-Level Air Traffic
Control Trainees. FAA Office of Aviation Medicine
Report No. AM-71-36, 1971.

. Davis, C. G., R. Kerle, A. W. Silvestro, and W. H.

Wallace: Identification of Training Requirements
in Air Traffic Control. FAA/BRD-40, Project 0,
Report 36, 1960.

. Davis, C. G., R. Kerle, A. W. Silvestro, and W. H.

Wallace: An Activity Analysis of the Positions in
a High Activity Air Traffic Control Tower. FAA/
BRD-40, Project 0, Report 39, 1960.

. Davis, C. G., W. H. Wallace, R. H. Kerle, and A. W.

Silvestro: Analysis of Controller Functions at the
High Altitude Sectors of the New York ARTC
Center. FAA/BRD-40, Project O, Report 35, 1960.

. Davis, C. G., W. H. Wallace, R. H. Kerle, and A. W.

Silvestro: Activity Analysis of Positions in a High
Activity ARTC Center. FAA/BRD-40, Project 0,
Report 37, 1960.

DuBois, P. H.: Multivariate Correlational Analysis,
New York, Harper & Brothers, 1957, 202 pp.

26

11. Guilford, J. P.: Fundamental Statistics in Psychol-
ogy and Education, York, Pennsylvania, Maple
Press Company, 1956.

Nagay, J. A.: The Development of a Procedure for
Evaluating the Proficiency of Air Route Traffic
Controllers. Civil Aeronautics Administration, Re-
port 83, 1949,

Nagay, J. A.: Field Tryout of a Procedure for
Evaluating the Proficiency of Air Traffic Controllers.
Civil Aeronautics Administration, Report 91, 1950. |

Sullivan, E. f., W. W. Clark, and E. W. Tiegs:
Manual for California Test of Mental Maturity;
Advanced Form, Los Angeles, California Test Bu-

reau, 1957, 39 pp.

Trites, D. K.: Problems in Air Traffic Management:
I. Longitudinal Prediction of Effectiveness of Air
Traffic Controllers. FAA Civil Aeromedical Re
search Institute Report No. 61-1, 1961.

12.

13.

14.

15.

16. Trites, D. K.: Problems in Air Traffic Management
VI. Interaction of Training-Entry Age With Intel
lectual and Personality Characteristics of Air

Traffic Control Specialists. FAA Office of Aviatior
Medicine Report No. AM-65-21, 1965.

Trites, D. K., and B. B. Cobb: Problems in Ab
Traffic Management: III. Implications of Age fo:
Training and Job Performance of Air Traffic Con
trollers. FAA Civil Aeromedical Research Insti
tute Report No. 62-3, 1962.

Trites, D. K., and B. B. Cobb: Problems in At
Traffic Management: IV, Comparisons of Pre
Employment, Job-Related Experience With Aptitud
Tests as Predictors of Training and Job Perform
ance of Air Traffic Control Specialists. FAA Civi
Aeromedical Research Institute Report No, 63-31
1963.

Trites, D. K., and B. B. Cobb: CARI Research o.
Air Traffic Control Specialists: Age, Aptitude, an
Experience as Predictors of Performance. FA:
Civil Aeromedical Research Institute, unnumbere
report, 1964.

1.

18.

19.


fronted in research aimed at progressive
improvement of personnel selection procedures.
It is commonly referred to as “the problem of
creeping criteria” and, inasmuch as officials re-
sponsible for training traditionally strive to up-
grade their programs, it is a problem that is
normally and hopefully anticipated.

The authors of the present report hypothesize
that the FAA has also followed much the same
pattern: that ATC training programs and per-
formance evaluation standards have been up-
graded; that the changes (the exact nature and
extent of which would be difficult to determine
reliably) are partially attributable to training
requirements associated with the increasing com-
plexity of the air traffic management system and
in part to the inherent tendency of trainitg
personnel to “customize” instructional material,
instruction, and/or subjective performance eval-
uation standards relative to the general level and
range of mental abilities characterizing the in-
coming clases. Moreover, the vast majority of
trainees recruited since 1963 have been of excep-
tional mental calibre and, thereby, comprise
groups which would be characterized by sub-
stantial restriction of range of aptitudes.

D. Rationale for Improving Aptitude Screen-
ing. There are a number of reasons why it may
be assumed that some improvement in the cur-
rent aptitude screening process might be
achieved. First, the air traffic management sys-
tem has undergone change in recent years, and
it is possible that the job tasks now require cer-
tain mental skills supplementary to those cur-
rently being assessed.

Moreover, the research in which the six CSC
tests were initially validated also yielded ap-
preciable validities for a number of other CSC
tests and several commercially-published copy-
righted instruments. The latter group included
several tests which apparently involved a
speeded perceptual-discrimination factor and
some which pertained to either coding skills,
comprehension, rapid integration or processing
of information, or memory. Although none of
these tests appeared in the group which the
multiple regression analyses indicated most use-
ful for selection purposes, the results led us to
suspect that the instruments were measuring im-
portant ATC-related aptitudes albeit inade-
quately. Such findings, supplemented by those
obtained for a number of different spatial tests,

were interpreted as illustrating the need for de-

velopment of tests “tailored” to fit the purpose.
While none of the various spatial tests in-
eluded in CAMI’s numerous experimental bat-
teries has failed to validate at statistically
significant levels, the validities of each have gen-
erally been much more variable from sample to
sample than for non-spatial tests. This also
applies to CSC Booklet 51 of the operational
screening battery. We concluded several years
ago that a “visual imagery” type of speeded
spatial ability was a major determinant of ATC
performance, but we were unable to locate an
instrument fully appropriate in this respect:
The development of a special test, the complexity
of which would have required a major researc
effort, was not undertaken due to other researc!
commitments having higher priorities. !
\

II. Procedure.

The Directional Headings Test should be con
sidered as a fortuitous outgrowth of develop,
mental efforts which were undertaken on
somewhat intuitive basis rather than a conse
quence of extensive background research, jol
activity analyses, and the like. It was develope
“in house,” within a period of only four week
by personnel of CAMI’s Psychometrics Uni
who interpreted the research findings discusse:
immediately above as indicating that the apti
tude screening process might be further im
proved through more adequate assessment of th
applicants’ speeded  perceptual-discriminatio,
and coding (or decoding) skills. In conceivin,
the type of task ultimately embodied in Part
of the DHT, the initial exploratory effort
focused upon the use of diagrams simulating
navigational compass. However, the pictori:
approach was abandoned on the hypothesis th:
if the data pertinent to the solution of each prol
lem were presented in a different format, suc
as described below, the test would be more difl
cult, and therefore provide better opportunit
for individual differentiation.

The DHT is a 3-part test in which the subje
js allowed a total of only 90 seconds for the soli
tion of 60 items, or problems, of each part. J
each problem, the subject is presented one, tw
or three “bits” of information relating to tl
cardinal points on a mariner’s compass. F:
example, the letter “N,” the symbol “/\,” ar


Training Marginal Non-marginal
Failures Passes Passes
Form B
Directional
Headings Sample 3A: Enroute ATCS Trainees of GS-7 Level
Test Score (D.H. correlates .44 with A+L Grade and .15 with P-F status) Nh
72
40 & > 78.3% 92
38
30 - 39 36.7% 67
14 7
2 & < 50.0% 25.0% 28
Sample 3B: Enroute ATCS Trainees of GS-9 Level
(D.H. correlates .50 with A+L Grade; P-F validity not computed)
41 7
40 & > 12.5 87.5% 8
3
30 - 39 100.0% 3
29& < 1 1
100.0%
Combined Samples 3A and 3B: Total of D.H. Test Form Examinees
(D.H. score correlates .45 with AtL Grade and .15 with P-F)
y 79
40 & > 79.0% 100
41
30 - 39 58.6% 70
14 8
29 & < 48.3% 27.6% 29

‘rcure 4, Proportion of Directional-Headings-Test Form B examinees (En Route only) within each of three
D.H. performance score groupings who failed Academy basic training, “marginally passed,” or passed with
grades represented in the approximate upper three-fourths of the A+L Grade distribution of passing Ss only.

\RTCC trainees, the 28 with DHT scores of 29
nd less included only 7 “non-marginal passes”
nd 21 who either failed or performed “margin-
lly.” For each of the higher DHT score in-
arvals, a progressively higher proportion of the
tS-7 subjects were “non-marginals.”

D. Screening Potential of the DHT Versus
te CSC Test Battery. As mentioned earlier,
erformance measures on the operational CSC
TC Aptitude Test Battery were received for
31, or only 55.8 per cent, of the 504 GS-7 ATCS
‘ainees who were experimentally assessed with

15

DHT Form A. Moreover, the 281 cases included
only 81 training-course failures. Inasmuch as
the 31 attritions represented only 11 per cent of
the sample of 281, whereas 102, or over 20 per
cent, of the 504 were known to have failed, the
sample should be regarded as biased. The rea-
sons why bias occurred in the collection of the
data are unknown. (Over 97 per cent of the
102 non-graduates, like 95 per cent of the 402
graduates, checked “Yes” to an item of the
CAMI Questionnaire indicating that they had
taken the CSC Battery in the process of estab-
lishing eligibility for appointment to training.)



csc
Test
Score

260 & >

230 - 259

2299 & <

Total

Form A
D. H. Test
Score

44 &>

34 - 43

33 & <

Total

44 & >

34 = 43

33 & <

Total

44 & >

34 - 43

33 - <

Total

44 &>

34 = 43

33 - <<

Total

Training Marginal Non-marginal
Failures Passes Passes
3 20
12 .07%98.0% 80.0%
4 10 24
10 . 574 26.3% 63.2%
3 “7 3 4
30 . 0%, 30.0% 40.0%
1) 15 48
13 7% 20.5% 65.8%
7, 8 32
20 .0% 80.0%
27 LL 48
18.0% 78.7%
7 12 16
20 . 0%; 34.3% 45.7%
9 31 96
. 22.8% 70.6%
i a 13
16 716. 86.6%,
2 “5 21
13.3% 16.7% 70.0%
7 11 9
25 . 9% 40.7% 33.3%
12 17 Uy 43
16.7% 23.6% 59.7%
65
81.2%
93
72.1%
29
40 .3%
187
66.6%

38

10

73

61
35

136

15
30

27

72

80

129

72

281

‘eure 6. Percentages of ATC trainees by CSC-Test-Score and Directional-Headings-Test-Score categories who
failed Academy basic training, passed with grades comprising the approximate lower one-fourth of the dis-
tribution of “A+L” training grades of the passes only (ie. “marginals”), or passed with grades in the upper

three-fourths (i.e, “non-marginals”).

The data pertain to 212 ARTCC trainees of Sample 1A and 69 TATC

trainees of Sample 2A, all of GS-7 level and for whom operational CSC-Test-Scores were forwarded to CAMI.

17



A PROPOSED NEW TEST FOR APTITUDE SCREENING OF AIR
TRAFFIC CONTROLLER APPLICANTS

I. Introduction.

' This report pertains to the development and
oxperimental validation of a new, highly-
speeded, and rather novel paper-and-pencil type
wptitude test and discusses its potential as an
additional screening device for improving the
validity of the Federal Aviation Administra-
‘ion’s (FAA) selection of personnel for training
is Air Traffic Control Specialists (ATCSs).
Che test, referred to as the “Directional Head-
ngs Test” (DHT), was developed by the Avia-
ion Psychology Laboratory of the FAA’s Civil
Aeromedical Institute (CAMI). In seeking to
stablish whether the findings justify operational
mplementation of the DHT, it is important first
o consider the nature and effectiveness of tests
‘urrently being used in the selection process.

The Current ATC Aptitude Test Battery

A. Validation of CSC Test Battery. Prior to
964, the FAA’s screening and selection of per-
onnel for ATC training generally involved no
ormal assessment of an applicant’s mental
bilities or aptitudes. Beginning in July 1962,
nd for 18 months thereafter, aptitude measures
‘ere considered in the selection of a limited
umber of trainees who possessed little or no
revious job-related experience. From January
964 through October 1968, a qualifying aptitude
idex, reflecting performance on a battery of six
ivil Service Commission (CSC) tests, consti-
uted a major eligibility requirement of al/ ap-
licants—regardless of their experience and
sher qualifications. The aptitude screening in-
ax was an outgrowth of extensive research
mducted by CAMI during 1961 through
363.4 18 19

Specifically, the six CSC tests were identified
rom among 27 experimentally administered
struments as yielding the best composite meas-
ve (of aptitudes) for prediction of training
formance. One of the tests, designated as

CSC Booklet 24, provided a measure of numer-
ical ability. Others were: CSC 51, which per-
tained to spatial relationships; CSC 135,
“Following Oral Directions”; “Abstract Reason-
ing” and “Letter Sequence,” both of which were
parts of CSC Booklet 157; and a special, omni-
bus test called “Air Traffic Problems” which the
American Institute of Research developed in
1950 for the CAA. The latter was ultimately
designated as “CSC 540.” In that research, the
CSC Battery was administered to a total of 893
men as they entered Academy ATC training.
The results of subsequent analyses included a
product-moment correlation of .51 between the
CSC composite scores and a variable which re-
flected an average of each individual’s examina-
tion grades and laboratory performance ratings,
and a point-biserial correlation of 40 between
the CSC measure and Pass-Fail status. More
importantly, an examination of the data to de-
termine a “cutting score” for screening purposes
revealed that 182 (or 67.1 per cent) of the 271
training-course attritions failed to attain CSC
scores of 190 or higher as compared with 222
(or only 87.5 per cent) of the 622 who success-
fully completed the course. However, in devel-
oping the new standards for implementation in
January 1964, officials chose to establish 210,
rather than 190, as the minimum qualifying
score. This relatively high screening hurdle was
instituted because records indicated that the
number of applicants had, for several months,
vastly exceeded the number of ATC trainee po-
sitions, and a further reduction in the recruiting
of ATC personnel was anticipated. As shown
in the upper portion of Appendix A, 576 (or
64.5 per cent) of the 893 trainees involved in the
validation study scored less than 210 on the CSC
Battery. Although 357 (or 62 per cent) of the
576 passed Academy training, 219 did not, and
the 219 represented 81 per cent of the 271 train-
ing failures within the entire sample. Selection





Training Marginal Non-marginal
Form A Failures Passes Passes
Directional
Headings Sample 1A: Enroute ATCS Trainees of GS-7 Level
Test Score (D.H. score correlates .38 with A+L and .24 with P-F status) N
119 ,
40 & > 71.7% 166
81
30 - 39 63.8% 127
29 & < 19 25 69
27 5% 36.2%
Sample 1B: Enroute ATCS Trainees of GS-9 Level
(D.H. correlates .50 with A+L; P-F validity not computed)
7
40 & > 77.8% 9
10
30 - 39
90.9% u
4
29& < 66.6% 6
Enroute Trainees of Combined Samples 1A and 1B
(D.H. correlates .37 with AtL Grade and .24 with P-F status)
40 & > 126
“2.0% 175
30 - 39 ea
65.9% 138
20 29
Wk <
26.7% 38.6% 75

‘GuRE 2, Proportion of En Route ATCS trainees within each of three Directional-Headings-Test-Score groupings
who failed the Academy’s basic ARTCC training course, “marginally passed,” or passed with grades compris-
ing the approximate upper three-fourths of the A+ Grade distribution for the passes only.

ailed to complete training; 19 (27.5 per cent)
aarginally passed with A+L grades represented
a the approximate lower one-fourth of the grade
istribution of the passes only, and that only 25
36.2 per cent) completed their course as “non-
aarginal passes.” In contrast, 63.8 per cent of
he 127 with DHT scores of 30 to 39 were “non-
rarginal passes.” The results for the 166 GS-7
.RTCC trainees who scored 40 and higher on
he experimental aptitude test are impressive but
nly slightly more so than those pertaining to
sainees who scored 30 to 39. No conclusions
an be derived on the basis of the results ob-

tained for the 26 ARTCC trainees of GS-9 level.
Only two of the 26 failed, three were “margi-
nals,” and the other 21 were “non-marginal
passes.” It is possible that the pre-FAA ATC
experience of these trainees enabled them to
master more readily the training materials than
their less-experienced and lower GS-rated class-
mates, but such a hypothesis should be viewed
as rather tenuous because the sample of En Route
GS-9’s was so small.

The results of corresponding analyses on the
data of the GS-7 and GS-9 Terminal students
were much more impressive than obtained for

13



Training Marginal Non-marginal
Failures Passes Passes
Form A
Directional
Headings GS-7 Level Enroute and Terminal ATCS Trainees For Whom
Test Score Operational CSC ATC Aptitude Test Scores Were Available N
11 65
44 & > 5713 8% 81.2% 80
Li 26 93
34 - 43 7 WZ, 20.2% 72.1% 129
17 26 29
33 6 < 23 .6%, 36.1% 40.3% 7”
1 63 187
Total 11.0 22.4% 66.6% 281
GS-7 Level Enroute and Terminal ATCS Trainees For
Whom No Operational CSC ATC Test Scores Were Available
Ly 38
44 & > 22.0% 76.0% 3°
19% 18 56
34 - 43 20 4% 19.4% 60.2% 93
41 15 24
33. & < 51.2% 18.8% 30.0% 80
71 34 118
Total 31.8% 15.3 52.9% 223
Total Group of GS-7 Level Enroute and Terminal ATCS
Trainees Experimentally Administered the Directional
Headings Test (Form A) Upon Entering the FAA Academy
103
44 & > 79.2% 130
14:
34 - 43 1% 222
41 53
33. & < 27.0% 34.9% 152
Total 8 5% 504

Freure 5. Distribution of training-course failures, “marginal” and “non-marginal” passes within each of thre
Directional-Headings-Test-Score groupings for 281 GS~7 level ATCS trainees (of Samples 1A and 2A) know
to have been operationally examined with the CSC-ATC-Screening Battery and for a similar group of 25
trainees for whom no CSC-Test scores were forwarded.

Notwithstanding the shortcomings of the sample,
the authors had no recourse but to deal with the
281 cases in analyses aimed at comparing the
screening potential of the DHT with that of the
CSC Battery.

16

The DHT Form A Scores of the 281 GS-
ATCS trainees for whom CSC Test data wei
available were arrayed from low to high and tI
resulting distribution was consequently divide
into approximate fourths. The lower fourt