Nickels, B.J., Bobko, P., Blair, M. D., Sands, WA. &
Tartak, ©. L. (1995, January). Sepensriex and Con-
wrel Hiring Assessment (SACHA) final job analysis
report, (CORL-OO7A delivered under FAA coa-
tact DTFAG1-91-C-00032). Washington, DC:
Federal Aviaticn Administration Office of
Personnel.

Poucay v. Pracential Insurance Ce., 668 E.2d 795 (SCA)
(1982).

Rock, D. B., Dailey, J. T., Ozur, H., Boone, J. O., &
Pickrel, E. W. (1984a). Validity and utility of the
ATC i tests battery. Study of Acad-
cary trainees, 1982. In S. B. Sells, }. T. Dailey, &
E,W. Pickrel (Eds.), Selection of air srafffic control-
devs (pp. 459 - $02). (DOT/FAASAM-84/2}. Wash-
ington, DC: Federal Aviation Administration Of-
fice of Aviation Medicine.

Rock, D. B., Dailey, J. T., Czur, H., Boone, J. O., &
Pickrel, E. W. (1984b). Conformity of the new
experimental test bextery o the Uniform Guide-
lines on Employee Selection Requirements. In S.
B. Sells, }. T. Dailey, & E. W. Pickrel (Eds.),
Selection of air sraffic conzreilers (pp. 503 - 542).
(DOTIPAA/AM-84/2). Washington, DC: Fed-
eral Aviaticn Administration Office of Aviation
Medicine.

is

Sackett, P. R., &¢ Wilk, S. L. (1994). Wichin-group
ae oe re oncen paebebogie,
vesting. American Prychologist,
ry 929 - 954.
Schmids, F. L. (1988). Tas problem of groop differ
ences in abilicy test scores in employment sehc
Son. Jouraal of Vocational Beier, 33, 272-292.

Sells, S. B., Dailey, }. T., & Pickrel, E. W. (Eds. 1984,
Selection of air traffic contvollerr. (OOTIFANAM-
84/2). Washingeon, DC: Federal Aviation Ad-
ministration Office of Aviation Medicine.

Seymour. R. T. (1988). Why pleinaffs’ counsel chal-
lenge cests, and how chey can successfully chal-
lenge the theory of “validiry generalization.” fexr-
nal of Vocational Bebavior, 33, 331 - 364.

Siegel, A. 1. (1978). Miniature job training and evaluz-
tion as a selection/classification device. Haman
Factors, 20, 189 - 200.

Siegel, A. 1. (1983). The miniature job raining and
evaluation approach: Additional findings. Persox-

nel Prychology, 36, 41 - 56.


Table 7

Reculls of sten-dcen hierarchical regression analysis, for test bias in research sampie by gender on
basis of corratation matrix without corrections for resiriction in range

Ansisysis Model RS ac aF F

Bacic v. Full: Overall bias TMC 0.837 339.947"
TMC + SEX + TMC_SEX 0.065 6oos = 38.6e°TM 140.01°°

Full v. Group: Slopes TMC + SEX + TMC_SEX 0.045 taa.oi*
TMC + SEX O.Nas 0.006 1@ x=»s50°°
TMC

Fall v. Crossproduct? TMC+SEX+TMC_SEX NIA

TMC + TMC_SEX
Basic v. Group: Intercepts TMC + SEX 0.085 29.50"
TTMC 0.037 0088 76.18°" 33e.94"°

Notes: "Pall v. rossproduct mode! comparitoe aot conducted. See Figure ? Sor logic and Gow of cieg-doen “5 < OT


Cemographic cheracteristics for reference population, all 1985-1022 FAA

Tamia 4

Academy entrants, end research sample
FAA Academy entrants
Reference Academy Sample
Characteristic Population extrams {N=9,552)
(N x 170,878) (14,392)
Sex
Male 132,708 11,460 7935
Female 37,870 2,932 1,817
Race
Asien/Pac-islard 91 3
Americen indian 195 131
African American Bis 283
Hispenic 52S 293
White 12,366 8555
Missing Data 396 231
Education
LT High School 404
High School 28,147 1,576 1,066
Some college 2414 778 5.35%
Bachelor's degree $8,583 4,745 3,035
Advanced degree 3.934 178 6
Missing Data 1,096 148 6
Age
Mian BO 28.77
sD 299 2.85

applicants.


So EE

accurately reflect cue differences in capabilities and
performance by gender. One recent analysis of che
ATCS job found that perceprual processes such as
visualization and scanning, are important worker re-
quirements (Nickels, Bobko, Blair, Sands, & Tartak,
1995). Another analysis suggested percepeual speed
and seasoning with numerical information were rel-
evant cognitive attributes to the controller job (Broach
& Axl, in preparetion}. An analysis of the abilities
required specilically for success in the Screen also
pointed towerd the visual-spatia! domain of abilities
(Gibb, Smith, Swindells, Tyson, Gierslzowski,
Perschaver, & Haney, 1991). These studies indicate
that there is at least some need to utilize abilities in the
visual-spatial domain in the performance of ATCS
casks. The construct validity study of the OPM tes:
battery conducted by Harris (1986} provided evi-
dence thar the MCAT, in particular, measured some
aspects of this domain of job-reizvant abiliries with ics
emphasis on che definition and manipukstion of the
perceptual field end reasoning with verbal end av-
meric information. There appecr to be subsle but
persistent sex differences in the visual-spatial abilicies
domain (Halpera, 7986; Voyer, Voyer, & Bryden.
1995}. Sex differences in the abilities measured by the
ATCS written apzitude test battery mighr explain, ix
part, the mean score differences observed on the
predictor TMC. Similarly, sex differences on visual-
spatial abilicies important to performance in che Screen
might similarly account for the apparent differential
prediction of Screen scores from aptitude scores.
Curvenr research being conducted under che FAA
Separation and Coarrol Hiring Assessment (SACHA)
project (Bobko, Nickels, Blair, & Tarcak, 1994: Uni-
versity Research Corporation, 1994) may provide
further daza elucidating the relationships between
gender, visual-spatial abiliries as measured by apti-
tude tests, and ATCS job performance.

ON

REFERENCES

American Educational Research Association, American
Beychological Association, & National Council
on Measurement in Education. (1985). Stansierds
fer educesional and pspchological cesting. (Sth Ed.).
Washington, DC: American Psychologica! Asso-
cation.

Anastasi, A. (1988). Prrchologicel Tecting (6th od.), New
York: Macmillan.

Antonie». Ward's Cove Packing Co., £90 US 642 (1989).

}. © (1998). Employing sir traffic controllers. In
“H. Wing & C. A. Manning (Eds), Selection of air
traffic conzroliees: & ts, and
public interest, pp. 7-12, (DOTIFAA'AM-S1/9).
‘Washington, DC: Feders! Aviation Administra-
tion Office of Aviation Medicine.

Bardetr, C. )., and O'Leary, B.S. (1952. A differential
prediction model ro moderate the effects of het-
erogenecus groups in personnel selection and clas-
sification. Persoxme! Poycheieges, 22, 1-18.

Bartletr, C. J., Bobke, P.. Mosier, $. 8., & Hannan, R.
(1978). Testing far faimess with a moderated
multipie regression strategy: An akernative to dif-
ferential analysis. Personnel Prychologise, 33. 233 -
242.

Boblo, P. (2983). Aa analysis of correlations corrected

Bobko, P., Nickels, B.J., Blair, ML DL. & Tartak, E. 1.
(1994, March). Preliminary internal ceport om the
cvoreas ssazzs of the SACHA woadel and task inter
connections: Volsme J, Unverim report delivered
under FAA contrac DTFAG1-91-C-06032).
Washington, DC: Federal Aviation Administra-
tien O Ree of Personne!

Broach, D., & Aul, ]. C. (in preparation). Amaijse: of the
ATCS 308 using she Position Anahsis Questionnaire
(PAQ). Manuscrips in preperation.


of aptitude and group membership (cross-producz
model). The cross-pzeduct model was then compared
to the group model; a significant change in R° indi-
cated intercept, as well as slope differences between
groups. If no slope differences were found, then the
cross-product mode! wascompared to the basic model;
2 significant change in R? indicated oniy intercept
differeaces between groups. The general logic and
associated SPSS syatax for the step-down hierarchical
regression analysis are illuscrated im Figure 2.

Technical feasibility

Restaiction in range, statistical power, and crite-
tion bias are considerations in evaluating the techni-
cal Sensibility of 2 cest fairness investigation that must
be explicitly consider-4 under the Uniferre Guidelines
(29 CER 1697.14,B.(8).(c) and (e); 29 CFR
1607.16.U). Both explicie and incidental resiriction
in range are recurrent problems in ATCS selection
research, as evidenced by the sample sizes and descrip-
tive statistics in Table 4. Variance in TMC for the
research sample was explicidly restricted in range due
te selection. Therefore, correlations bereeen TMC
ard the SCREEN criterion were corrected with re-
spect to the reference population of 170,578 appii-

canes, using the formula presented by Ghiselli,
Campbell, end Zedeck (1981, p. 299). Correlations
berween variables indicating gender and the critesion
were incidentally restricted in range. These gender-
criterion correlations, including the gender-by-pre-
dictor croseproduct tc SCREEN correlation, were
cozrected with respect to che reference population of
170,578 applicants using the Ghiselli, Campbell. and
Zedeck (1981, p. 304) formula for incidental range
cesteiction. Finally, values for the population correlz-
tions berweea geader and the gender-predictor
srossproduct were computed. The overall structure of
the correlation matrix is described in Table 5; sample
and corrected correlations are presented in Table 6.
Sample correlations, without corrections for reseric-
tion im range, are resented in the lower left-hand
corner, while correcied and population correlations
fe presented in che upper right-hand corner of the
overall matrix. Separate differential prediction analy-
ses were conducted on the basis of sample and cor-
recesd correlations, as required by the Linsferm Guidelines
(29 CFR 1607.15.B.(8)).

Sample sizes im these analyses were clearly of suffi-
cient size to provide more than enough power to
decect even small statiztical effects. We estimated the

TC SEX TMC_SEX SCREEN
TMC yr, i i
SEX fr lp 7
TMC_SEX ts F, 5
SCREEN ls 7, rs

Mote: Sameple correistion matrix strecaure shown below the diagonal, comected
matrix stroctie above the Giagorai. 7, = sample cormlation: #5 = pooeiztion
correianion, where popoicion is erfenmace gogeiation of ail applicants r, = sample
correlesion corrected for eenticit restriction in range, based an reference
population of all applicants; ; = sample corvelution corrected fer incidental
restriction in range, based on seSesence populatios of si amplixaats


Table 2
Mean predictor and criterion score differences by gerker

Variable Group ro M SD SE t gf
TwC Males 132,708 T4 as 147 639 61.75° 170,576
Females 37,870 $9.32 14.37 6.074
SCREEN Males 20,252 T2795 1170 116 $si 12,754
Femmes 2,804 70.26 12.06 8.240
“p< O01
Table 3
Adverse impact analysis for refensnce pop. by gender for based
on OPM eligibiity codes
Sex
QPM Eligibility Males Females Row towls
Eligible 9.056 14,564 2,070
(S2.08E) (38.46%)
Failed test 49,502 20,077 9579
G7.0%) (53.62%)
Other ineligible 13,758 3,229 16,979
(10.36%) 6.53%)
Column totals 132,708 37,870 170,578



Figure t
Precketer (TRC) score distribution by gender for reference scpulatian of
applicants (N= 770,578)

Male (,000)

Famate (,000)

24

{eMeles > Females}

20

16

12

Selection ratio

The results of the adverse impact analysis by gen-
der, based on OPM eligibility codes, are presented in
Table 3. Approximately half (52.04%) of the 132,708
men were considered eligible for employment an the
basis of cheir aptitude test scores. With a majority
selection rate of about 50% and mean differences of
.35 SD, we anticipated a selection rate for women of
about 35 to 38%, based on Sackett and Wilk (1994).
In fact, 38.46% of the 37,870 women were eligible for
employment on the basis of the aptitude test scores.
The proportion of women determined to be eligibic
for employment was significaniy fess chan the pro-
portion of men (Z «= -46.63, p < .00!); the ratio of
female to male selection rates was .74. Using the 4/5ths
rule of thumb of the Uniform Guidelines, it appeared
shat use of scores on the written ATCS aptitude resx
battery to determine eligidilicy for employment re~
sulted int scariscically significant adverse impact against
female applicents.

DIFFERENTIAL PREDICTION ANALYSIS
Given the finding that there appeared ro be adverse

impact against women, the Uniform Guidelines (29
CFR 1607.14.B.(8).(0)) and Standards for Educational

anal Prychelogical Testing (Standard 1.20, p. 17) re-
quired an investigation of che relationship berweca
test scores and job performance for evidence of differ-
ential prediction by subgroup. We hypothesized thar
there was no difference in the predictive validity of che
test battery by gender.

METHOD

Sample

The differential prediction analysis was based on 2
san.ple of persons actually hired by the FAA on the
basis of their aptitude test scares. Between October
1985 and January 1992, a total of 14,392 ATCS
candidates entered the FAA Academy, The majority
(11,405) had competed under civil service regulations
fr hire and were entering the Academy for the first
time. Complete gender, racial idencification, predic-
tos, and criterion data were available for the evscarch
sample of 8,842 male and female stadents. There were
7,332 (82.9%} men and 1,510 {17.195} women in the
sample. Demographic information for all Academy
entrants and the research sample is preseated in Table
1. As with the reference applicant population, gender
(SEX) was coded as 0 for maies, and 1 for females.


Broach, D., & Manning, C. A. (1994). Validigy of the air

sraffic cansrel specialise nowradiar screen ase predicser

of in redar-besed atv traffic control

training. (DOTIFAASAM-94/9). Washington,
DC: Federal Aviation Administration Office of
Aviation Medicine.

Brokaw, L. D. (1984). Easby research on controller
selection. in S.B. Seils, }.T. Dailey, & EW. Pickred
(Eds.), Seleesion of ect traffic contraers, pp. 35-78.
(DOTIFAAIAM-84/2), Washington, DC: Fed-
eral Aviation Adminiscration Office of Aviation
Medicine.

Cohen, J. (1588). Sessa! poser anata: fe the bebe
jor sciences (2nd Ed.}. Hillsdale, NJ: Edlbaum.

Coben, J., 8 Coben, P.(1975). Applied mahiple regres.

Collins, W. E., Boone, J. O., & VanDeventer, A. D.
(1984). The selection of air traffic control special-
ist: Contributions by the Civil Aeromedical Insti-
ture. In SB. Sells, J. T. Daiiey, & EW. Pickrel
Eds), Selection of air traffic consreillers. 99. 79 -
112. (DOT/FAAVAM-S1/9). Warhingtoa, DC:
Federal Aviation Administration Office of Avia-
tion Medicine.

Della Roceo, P. S., Manning, C A, & Wing, H.
990) Selection of controllers for autemaied sy

from curvent research. (DOT!

FAA/ARE-50/13). ‘Washington, DC: Federal Avia-

tien Administration Office of Aviation Medicine.

Dunbar, S. B., & Novick, M. R. (1988). On predicting
success in training for men and women: Examples
from Marine Corps clerical specialties. fowrne! of
Applied Psycholegy, 73, S45 - $50.

EEOC ». Grephound Line:, 635 E,2d 188 (3CA) (1980).

Ford, J. K., Kraiger, K., & Schecuman, S. L. (1986).
Suudy of race effects in objective indices and sub-
jective evaluations of performance: A meta-analy-
tis of performance criteria. Phychodegica! Bulletin,
QQ, 336 - 337.

Federal Aviation Adminstration. (1993). FAA diversity
plex. Washington, DC: Office of the Administra-
wor for Hamen Resources Management, Career
Syseems Division.

Ghiselli, E., Campbell, j., & Zedeck, S. (1981). Afea-
surement theesy for the behavioral sevences, San Pran-

Cisco: Freeman.

te
“4

Gibb, G. D., Smith, M. L., Swindelis, N., Tysoa, D.,
Gierahowski, M. }., Petschauer, K. j., & Haney,
D.UL(1992), The deurlopment of an experimental
selection west baztery far cir orsffic consrol specialists.
Daytona Beach, FL: Embry-Riddle Aeronautical
University Airway Simulation Laboratory.

Gorrfredson, LS. (1994). The scieace and politics of
race-norming. American Psychologist, 49, 955 -
963.

Halpern, D. F. (1986). Sox differences in cognitive abili-
wes. Hillsdale, NJ: ésfbaum.

Hartis, P. (1986). A consernct validity stuiy of the Federal
Avistion Adminisration Consvoller Apzi-
aude Tet. Washington, DC: U.S. Office of Per-
sonne] Management Office of Seaffing Policy.

Hazeiwood Scizeol District ». United Scates, 433 US 299
(1977).

Laurenschlager, G. J.. & Mendoza, J. L. (1986). A step-
down hierarchical multiple regression ansiysis for
examiaing hypotheses about zest bias in predic-
tion. Applied Prychotegion! Measurement, 1D, 133 -
139.

Lee, R., Miller, K,, & Graham, W. (1982). Corrections
for restriction of and amenuation in crite
rion-related validetion studies. journal of Applied
Paychology, G7, 637 - 639.

Manning, C. A. (199%). Procedures for selection of air
traffic cantrol specisliss. In H. Wing & CA.
Manning (Eds), Sclecsion of air traffic controllers:
Complacsty, reguincments, end pebixe interest, pp.
13-22 (DGTIFANAM-9 1/9). Washington, DC:
Federal Aviation Office of Avia-
tion Medicine.

©. A, Della Rocco, P. S.. & Bryant, K.
(1989). Prediction of success in air traffic costrel
Sield training as « fxction of celection end screening
performance. (DOTIFAAIAM-89/6), Washingwoa,
DC: Federal Aviation Adminincaion Office of
Aviation Medicine.

Millsap, R. E. (1988). Sampling variance in the corela-
Gon coefficient under range restrictinn: A monte
casio srady. Joeraal of Applied Prycholegy 25, 456



Tabie 4
Descriptive siatistics for reference population of jcb epplicants, all 1985-1992 FAA Academy

entrants, and reasearch sample

Reference population (N=170578) All Academy entrants (N21 4,392)

Reseesch sample (N=95$2)

Variddle x SD Min Max M SD Min Max a SD Mie Max
Tc 73.30 1437 19.53 100.00 91.08 $43 70.06 100.05 Sis$ $03 MOS 100.00
SEX On @42 OD 040 el? O38

TMC_SEX 1S39 33.60 60.00 100.00 16.18 3480 00 100.00 18.52 343 00 100.00
SCREEN 7226 9385 2716 47 FEOS 2436 2716 STSD

Noss: — Screea score not applicable for reference popullatice of job applicazas



DOTHAMAM-96N3 Differential Prediction of FAA
Academy Performance on the
Basis of Gender and Written
pe AirTeeffic Control Specialist
GMee of Avistion Medicine

Washington, B.C. 20551 Aptitude Test Scores

Willie C. Young
Dana Broach
William L. Farmer

Civil Aeromedical Insticute
Federal Aviation Administration
Oklahoma City, Okishoma 73125

April 1996

Final Report
SCR SSE)
Dermeenee ae oraet J

This document is ..uiable to the public
through che National Techaical information
Service, Springfield, Virginiz 22161.

4 fava & U.S. Department
i 9605 yh 076 of Transportation
i Federal Aviation

iministratier
AGministretien DONC QlauiTY isoresy &



available statistical power using Coben's (1983) re
gression power tables (Tadle 9.3.1) for as many as 3
independent variables at an alpha of .63. The tisk of
a type Ill error (failing to find an effect chat in fact was
present) was very low, with 4.98 probability of derect-
ing even very smal} effect sizes (F < .03) with a sample
of more than 8,000 cases.

Finally, as noted by Sacket: and Wilk (1994),
Lautenschlager and Mendoza (1986), as well as by the
Uniform Guidelines (29 CFR 1607.16.U}, the feasi-
bility of an assessment of technical fairness depends
upon the quality of the job-relevant criterion: If the
criterion was systematically biased against women, for
example, then the regression-based method could not
be used to determine the presence or absence of
differential prediction by subgroup. The distribution
af cyiterion scores by gender is shown in Figure 3.
Observed mean secre differences in SCREEN by
gender were about 0.23 SD for the research sample.
These results are somewhat less than che estimated
differences of .3 to .4 SD by race reported by Ford,
Keaiger, and Schectman (1986). Thee are several
possibilities for these observed differences by gender:
these mean criterion differences may represent some
degree of “systemazic bias” agzinst women; the secm-
ing bias may have been confounded, at Jeast sn part,
with differences attributable to selection: these differ-
ences may reflect true distinctions in performance
that were incidenral ro sex; and the apparent bias may
be due co differences in information processing strat-
egies used by the sexes. However, the limits of the data
available for this srudy did not permit a definitive
evaluation of these alternatives. Therefore, we cannot,
with certainty, claim an unbiased criterion. Yer, in
accordance with the Uniferm Guidelines, Screen score
was properly measured, and relazed to subsequent
orge..lzationally valued outcomes. It was also no more
biased chan measures used in previous published se-
lection test fairness studies. Therefore, it was an ap-
propsiate criterion in this assessment of technical
fairness under the Uniform Guidelines.

RESULTS

Without corrections for reateiction in range

The adverse impact analysis suggested that use of
TMC to determine eligibilicy for eenployment as an
air traffic controller may have contributed to a sraa-
tion of adverse impact against women. The focus ofan
evaluation of technical fairness, therefore, shifted to
the degree to which the predictor score differentially
predicted the criterion. Semple correlations, without
corrections for restriction in range. are presented in
the lower left-hand eriangle of the matrix in Table 6.
TMC was significantly correlared with final score in
the Academy Screen (r= .1844, p < 001) and slightly
with the predictor-group crossproduct (r= .0296, 9<
.02). Gender (SEX) was cegatively correlated with the
criterion SCREEN score {7 = -.0847,p < .001), where
gender was coded as 1 for females and 0 for males. The
results of the differential prediction analysis using the
step-down hierarchical regression analysis on the basis
of the sample correlation matrix without any correc-
sions for restriction in range are presented in Table 7.
The gull hypochesis that = common regression fine
provided the best fic was rejected in the frst analysis,
suggesting the presence of some degree of test bias.
The increment in R* gained by using the fall modef
(predictor, group membership, and crossproduct).
rather than the basic mode! (predictor only}, was
significant (A A’ = 008, A F= 38.60.p ¢ 001). Next,
the aull hypothesis of same slopes by geader could not
be rejected; the subgroup model (predictor and group
membership) did not explain any less variance than
the full model (A F = 6,4 F = 1.02, 2s). Following the
analytic logic illustrated in Figure 1, the basic and
subgroup models were next compared to determine if
the intercepts were different for men and women. The
null bypochesis of same incercepts was rejected, with
removal of SEY leading to a significant reduction in
the ammount of explained variance (4 F = -.008,4F«
76.18, 9¢.002). Overall, che results obtained with the
uncerrected correlations indicated significant inter-

cert differences, but no differences in slopes by gender.


22 EE REE

Step-down hierarchical regression analysis logic and SPSS syntax

Agure 2

es

Rav BSCMOOR TET

Change in FF
‘ erga nn SF



Measares

Peedictars. TMC was used in our differential pre
diction znalyses as the measure of candidate aptivude
as it provided s measure of ability unadjusted for
previeus experience and/or military service. Descrip-
tive statistics for the predictor scores are presented in
Table 4 for the reference population of applicants, all
Academy entrants, and the research sample.

Criterion. The criterion in the differeatial peedic-
tion analysis was performance in the FAA Academy
initial ATCS craining program, known as the ATCS
Nonradar Sereen (“the Screen”). Training may be
used as a criterion measure where success in training
is “properly measured,” and the relevance of the
training can be demonstrared through comparison of
training content to critical or imporceat job behaviors
or by showing that training measures are related to
subsequent measures af job performance (29 CFR
1607.14.B.(3}). The Screen was originally established
in response to recommendations made by the US
Congressional House Con.mittee on Government
Operations (US Congress, 1976) te *... provide early
and continued seceening to insure the promps elimi-
nation of unsuccessful trainees and relieve the re-
gional facilities of much of this burden” (p. 13). The
Screen was based upon a miniavurized ersining-test-
ing-evaluation persoanel selection model (Siegel, 1978,
1983; Siegel 8¢ Bergman, 1975) in which individuals
with ac prior knowledge of an occupation are wrained
and then assessed for their potential to succeed in the
job. Performance in the Screen has been shown to
predict subsequent performance in tadar-based train-
ing ! co 2 years after entry into the occupation (Broach
& Manning, 1994) 2s well as completion of the
zigorovs on-the-job training sequence and ceriifica-
dion as a qualified “full performance level” coneraller
(Della Roceo, Manning, & Wing, 1990; Manning,
Della Rocco, & Bryant, 1989).

Thizteen assessments of performance, including six
classroom tests, observations of performance in six
laboratory simulations of non-radar air traffic con-
trol, and a final written ezemination, were made
during the Screen (Delia Rocco, Manning, & Wing,
1990). The fine! sunmed composite score (SCREEN}
was weighted 20% for academics, 60% for laboratory
simulations, and 20% for the final examination. A

Ww

minimum SCREEN score of 70 was required to pass
she Academy program. This final compasize score was
che crizerion measure in this study. Descriptive statis-
tics Cor SCREEN scores are alse presenred it, Table 4
for ali Academy entrants and for the research sample.

Brocedure

The classical, regression-based mode! of rest bias
was used as our analytic framework to evaluate the
degree to waich the written ATCS vese battery differ-
entially predicted performance in the Screen. A step-
down hierarchical multiple regression analysis
(Lautenschlager & Mendoza, !986) was used to evalu-
are test bias. The step-down approach evercomes the
shortcomings of the various step-up procedures
(Bartlett, Bobko, Mosier, & Hanna, 1978; Cohen &
Cohen, 1975} by accounting for che various changes
in the sun of squased ezror term incrementally, white
at the same time ensuring more statistical power than
the other methods (Lautenschiager & Mendoza). Step-
down analysis assumes the null hypothesis that a
common regression line provides the best least-squares
fiz to the data. The alternative is that 2 fall model,
including slope and interepe differences berwren
groups, is required to provide a significantly beter fiz
to che data.

Ous step-down analysis was conducted as follows,
using the SPSS (SPSS, Inc., 1989} regression proce-
dure. Firs, SCREEN was regressed on TMC only
(basic model). Second, the criterion was regressed on
TMC, the dummy coded group membership variable,
and the cross-product of TMC and chat dummy-
coded variable (full moded}. This full model was tested
against the simple model of criterion and predictor
test only for an incremental change in the # (good-
ness-of- fit index). A significant change in & sug-
gested pocential biss and dictated chat farther vesting
for slope and/or inzercept differences for the groups be
done. Third, to cest for slope differences berween
groups, SCREEN was regressed on TMC and the
durnmy-coded variable indicating group membership
{group mode}, and compared zo che full model. A
significan: increment in the &’, based on a compari-
soa of che group co full model, implied different
slopes. Fourth, if dope differences were found, then
SCREEN was regressed oa TMC and the cross-product


METHOD

Sample
The adverse umpect analysis was based oa determi-

nations of eligibility for employment made by OPM
for job applicants on the basis of a composite of two
written test scores. Over 200,000 job applicants have
taken the OPM written ATCS rest bareery since 1981.
Records for 179,578 applicants with complere test
scores wece available in the daca base. These records,
as provided by OPM, inchaded test raw scores, gender.
education, and 2 determination of eligibilicy for em-
ployment based on west scores; racial ideatification
dara were aot available. Demographic characteristics
for this reference population of applicants are pre-
seated in Table 1. Gender (SEX), as indicared by
OPM values, was recoded as 0 for males, and 1 for
females.

Selection test scares

The selection test scoze used by OPM to determine
eligibility for employment was a composite of scores
earned on cwo written ATCS aptitude tests: the Mul-
tiplex Coneroller Aptivade Test (MCAT), and the
Abscrace Reasoning Test (ABSR)}. The development,
psychometric characteristics, and validity of chese
written ATCS aptitude tests have been extensively
described elsewhere (Brokaw, 1984; Collins, Boone,
& VanDeventer, 1984; Manning, 2991; Sells. Dailey,
& Pic! rei, 1984}. Scoring of the tests was done ini-
tially by summing the MCAT (weighted 2) and ABSR
{weighted 1) scores. The resulting weighted scores
were then transformed via an OPM text score crans-
mutation cable into che Trensmuced Composite Score
(TMC). Abouc half of al epplicants were expected to
score a¢ or above the mean on this compesite (Rock,
Dailey, Ozur, Boone, & Pickrel, 1984a}. Applicants
with 3 years of general experience, 4 years of college,
orany combination of education and experience equat-
ing to 3 years of general experience and without prior

aviation expetience, were required co carn 2 TMC of

at lease 75.1 ro be eligible for employmeni. Applicants
with specific air eratfic controt-related aviation expe-
rience, or 4 years of college plus 1 year of graduate
study, were eligible for employment if they earned a
TMC of at least 70. In other words, acutscore of 75.1

be

oc 700n TMC, depending on applicant background,
was used to determine eligibility for employment.
Applicants not meeting these criteria were ineligible
for employment as controllers. The determination of
eligibility for employment was made by OPM. Codes
indicating chat an applicant had cither failed the test
CIA’) or scored tow low for consideration (‘1S’), based
on TMC, were recoded as vest failures. All other
ineligibility codes were recoded ss “other ineligible,”
and codes indicating eligibility were recoded as “eli-
gible” for employment. The adverse impact analysis
wes based on this eligidility variable.

Procedure

The adverse impact analysis was conducted in two
steps. Firs, TMC disteibutions were analyzed by
gender, @ etese was used to evaluate mean score
differences. Second, selection rates ou the basis of
eligibility codes by gender weie evaluated. The pro-
portion of applicants determined to be eligible on the
basis of theiz test scores was compared ro the prepor-
tion ruled as inctigible on the busis of test scores;
applicants determined to be ineligible on any other
basis (e.g., age, salary requirements, experience, of
education) were excleded from the analysis. Fisher'sZ
rest was used to statistically compare selection rates.

RESULTS

Group differences

Analysis of group differences in predictor scores by
gende: are presented in Table 3. Males earned signifi-
cantly higher mean TMC scores (4f = 74.44, SD =
14.27) than femaies (Af = 69.32, SD = 24.37;
KITO,5S76) © 61.75, ps 001}. The distribucion of
TMC by sex im the research sample is illustrated in
Figure 1. The standardized effect size (d) for gender
on TMC scores is 0.35 SD. corresponding to 2 small
to medium effec: sige (Cohen, 1988). This contrasts
with previous research suggesting thet mean differ-
ences in TMC were insignificant (Rock, Dailey, Ozur,
Boone, & Pickrel, 19846). Mean score differences
might be expected to translace into differences in
selection rates by gender. In the ATCS selection
process, only candidates scoring at or above the aver-
age TRIC were aligibie for employment.


Wieh corrections for restriction iz range
However, as shown in Table 2, the semale range of
scores on the predictor wes severely restricted: 25 a
consequence, evidence based on those uncorrected
correlations may be somewhat misicading as ta the
Ssirness of the prediccor (29 CFR 1607.14.B.8.{c}}.
Analyses based on correlations corrected for explicic
and implicit restriction in range may provide a better
assesament of the fairness of the OPM rest battery
with respect to the applicant population. Population
and correceed correlations are presented in the upper
zight-hend siangle of the matrix in Table §. The
estimated pepulation cerrelation betwen TMC and
performance in the Academy Screen increased ftom
-1844 to 4724 with correction for eaplicit restriction
in range. After cotrecting for incidenzal restriction in
sange. the correlation between gender and SCREEN
decreased ro -.0681, es did the correlation between the
cressproduci and SCREEN (-.0408). The results of
the differential prediccion analysis, using che step-
down hicrarchical regression analysis on the basis of
the corrected correlations, ere presented in Tab‘e 8.
The null byporhszis of a common regression line was
rejected, suggesting the presence of some degret of zest
bias The increment in R° associated with the full
model over the basic model wes significant (& =
0187, Fs 117.76,p5 001). Next. the auil hypothesis
of same slopes by gender was rejected; the subgroup
model (predicror and group membership} expiained
less variance than the full model (F = -.0187, Fs
235.34, ps 001). Following the analytic logie Hlus-
teated in Figure 1, the full and crossproduct models
wert next compared to determine if the intercepes
were different for men and women. The null hypoth-
esis of same intercepts was also rejected, wich temoval
of SEX leading to a significant seduction in the
amount of explained variance (R= -.0181,F= 228.41,
Ps 001). Overall, the zesults obtained with the
correceed correlations indicated the aced for separate
regression equations for men and women for predic-
tions based on both caw and standardized predictor
score. Therefore, correlations berween TMC and
SCREEN were computed for mex and womes sepa-
rately, corrected for explicit restriction in range based

ys

on the stendard devistion of the apricude by sex
CFMC_SEX} inceraction term, and submitted to re-
gression enalysis. The equation for men was:
SCREEN’ = -24.8689 + (1.0997 * TMC)
compared to and equation for women of
SCREEN" = -23.2254 + (1.0102 ° TMC}
where SCREEN’ is the predicted score in the FAX
Academy ATCS Nonrader Screen. The regression
equarions are plowed in Figure 4.

DISCUSSION

Overcll, the analyses cepoczed in chic study indi-
cated thar the written ATCS aptitude test battery did
not fulfill the cechaicel fairness requirements oudined
by the Uniform: Guidelines on Exaptepee Selection Pro-
ceduves. The results of the advecse impact analysis
indicated that use of the weighted composite of MCAT
and ABSR scores as 2 qualiScation criterion resulted
in the exclusion of greater proportions of women than
men from further consideration for employmens.
Moreover, the adverse impact could be strribared to 2
specific practice (Amtesio x, Ward's Cone Packing Co.,
1989, EEOC eo. Grephoumd Lines, 1980, Pounce v.
Prudential Insurance Co., 1982) and was stetisneally
significant (Hasriwoad Schsel District . United Seetes,
2977}. Similacly, there appeared to be subtly differzat
relationships for the sexes between apcitude score and
sebsequent performance ar the Academy, after cor-
recting the sample daza correlaiians for restriction in
range. The corrected majority regression line slightly
wverpredicted the performance of the minority group,
as shown in Figure 4. Schmids (1988) suggested thar
this is a common firding in differentia! prediction
analyses. For example, Dunbar and Novick (1988}
reported timilar results fer predictions of ercining
success from the Armed Services Vocational Apcitude
Bactery (ASWAB) scores by gender.

Evidence for differentia) prediction such as we
found ia this study of the ATCS wrinen apticude test
battery has been discounted oa rhe basis of fecrors
such as use of inappropriace statistical procedures and
defects in study desigus (Heater, 1973}. On one
hand, the statistical effects devected in our differential


Figure 4
Regress n functions by gender in research sample

prediction analyses were generally small and detect-
bie only with very lerge samples, after corrections for
restriction in range. One might argue that, asa conse-
quence, the resules were artifacts of en inappropriate
analysis of corrected correlations, and have litele prac-
tical significance. We would counter by noting that
corsected correlations may, in fact, provide more
accurate estimates of test validity, particularly in large
samples and under scringent selection ratios (Robko,
1983; Millsap, 1988). Uncorrected coefficients ap-
pear to be downwardly biased estimares of the true
population validity coefficients (Lee, Miller, & Gra-
ham, 1982), Therefore, differential prediction ancly-
ses based on corrected correlativns chat provide less
biased estimates of true population values are likely to
provide similasly less biased estimazes of population
effects, and are not artifecmal. Moreover, we believe
thar chese effects cannot be lightly dismissed in vier
of the very real practical consequences for the ATCS
selection program. One practicai consequence of 2
mean score difference on TMC < (9.35 SD was ad-
verse impact on women, asdefine, under the Liziform
Gaideline:. Moreover, the practical consequence of
the apparent differential prediction in the population
was chat women mar have effectively needed 2 higher
TMC chan men to have an equal likelihood of passing

the FAA Academy. The implications of differential
prediction relative co the consequences of over-
prediction will be investigated in greeter detail in
another study.

On the other hand, unmeasured variables may have
been confounded with the predictor, resulting in 3
defective study design (Anastasi, 988). One might
suspect, for example, thar education end seoceson the
aptitude test might be confounded in view of the
generally positive cortelation between such rests and
educational attainment: the group with lower scores
on an aptitude zest bartery might have lower overall
educational leveis than the other group. However, 2
significantly greater proportion of women (39.3% of
34,479} than men (35.7% of 118,735) had achieved
a bacesiaureate degree or mote in the reference pepu-
lation of 170,578 applicants (Z = 12.22, p 5.001). A
similar pattern was found for the sample of 9,552 FAA
Academy entrants used in che differential prediction
analysis. These dats provide some evidence to suggest,
pending more detailed analyses, that unmeasured
variables such as education may not account for the
observed differential prediction in this study.

There is an alternative explanation to conclusions
of rest bias or artifactual cesults due to statistical
surtections ct unmeasured variables, The results right


DIFFERENTIAL PREDICTION OF BAA ACADEMY PERFORMANCE ON
THE BASIS OF GENDER AND WRITTEN AIR TRAPFIC CONTROL
SPECIALIST APTITUDE TEST SCORES

The Federal Aviation Administration (FAA), ia iss
1993 Diversity Plan, made 2 commitment to attract,
rerein, develop, and manage a diverse work force that
visibly reflected the American population at large by
che year 2000. Achieving this goal will require sub-
stantial changes in the demographic profile of the Air
‘Traffic Control Specialise (ATCS) occupation, che
single largest (17,000) and most publicly visible occu-
pational group in the agency. Air traffic contro! is 2
career field in which fernale workers have been histori-
cally uader-represented relative to the American popu-
lation at lange. Entry iate the occupation hes deen
derermined since 1983 by applicant performance ona
written apticude test bartery administered by the US
Office of Personnel Management (OPM; Aul, 1991).
This test barrery emphasized the crganization, defini-
dion, an2 manipulation of the perceptual field through
verbal and numeric reasoning (Harris, 1986). Yet, it is
exactly such a test bartery of cognitive abilities that
may have beea an inadvertent device for the exclusion
of women from this rraditionally male occupation.

Our purpose in this paper was to examine the
technical fairness of the written ATCS aptivade test
battery as the first step toward assessing to what
degree, if any, that the bastery may have served as an
“engine of exclusion” (Seymour, 1988) of women
from: the ATCS occupatior.. By technical fairness, we
are referring to the regression model of vest bizs for
which there is 2 reasonable professional consensus, as
embodied in the 1985 Stenderds for Educational sxd
Pryehbological Tessizg (American Educational Research
Association, American Psychological Association, &
National Council on Messurement in Education},
rather than a socially constructed stendard regarding
rese use (Sackest & Wilk, 1994; Gorrfredson, 1994).
Technical fairness in this sense, and under the Usi-

form Guidelines on Eneployes Seiection Procedures (25

CER 1607), encompasses two issues. First, the impact
on protected groups arising from use of s particular
eet score on the predictor must be evaluated. A
selection rate for any protected group chat is less than
four-fifths (4/5 or $046) of that of the majority group
will “.. generally be regarded by the Federe! enforce.
mentagencies as evidence of adverse impact” (29CER
1607.4.D}. Second, where use of 2 selection proce-
dure results in adverse impact, the Uniform Guidelines
require that the rest user evaluate the degree to which
differential predictions of future job performance are
made from selection test scores by subgroup (29 CFR
1607.14.B.(8).(b)). This study investigaved the tech-
nical Gsirness of che ATCS written aptitude test bar-
tery toward women from iwo perspectives: adverse
impoct and differential prediction.

ADVERSE IMPACT ANALYSIS

Previous research on written ATCS selection tests
suggested chet mean score differeaces by gender were
insignificanz (Rock, Dailey, Ozur, Boone, & Pickrel,
19842, pp. 476) end that, overail, “the evidence for
adverse impact against women based on this sample
‘was marginal, at best” (Rock, Dailey, Czuz, Boone. &
Pickrel, 1984b, pp. 507). This conclusion was based
primarily on results of cheir 1984 study in which $7
of men (2 = 3835} passed che screen in comparison to
45% af women (a = 1473}. The adverse impact ratio
in this case was 6.78 rather than the 0.80 required
under the “four-fiths rule of thumb.”

In the presen: study, we hypothesized that the com-
posite of scores earmedon the written ATCS apdtude test
Satrery, as used by OPM ro derermine eligibility for
employment, had a0 adverse impact on women appl
cants. The composite of test scores would be considered
technically S:ic if chere was no adverse impact arising
from. ies use as a personne! selection device.


Tecnica! Bocumenionon age

TS Report No. @. Gowgmenent Acceston No | $ Recipient's Cotciog No |
DOTIFAAAM-96/13
& tie ond Satta & Report Date i
Differential prediction of FAA Academy petforance on the basisef gender | April 1996 i
| and wrieven Air Traffic Control Specialist apticade test scores LL H

i "6. Partommng Cres~ation Code :

2. Authors 8. Performing Orpanzation Report No.
Young, W.C., M.Ed Broach, D., Ph.Ds and Farmer, W.L., MS.

i FAA Civil Asromedical Institute
P.O. Box 25082
Oklahoma Cary, OK 73125

40. Work Unt No CRASS)

1, Contact or Grant ho.

12. Sponsonng Agarcy Name end Address 1S. Type of Recort and Penod Covered
Office of Aviation Medicine

Federal Aviation Administration

800 Independence Avenue, S. W.

Washingron, DC 20591

1S. Supplementary Notes

This research ws conducted under task AM-92-3-HRR-128

1& Abshoct

‘The technical fairness of the writren air traffic control specizlise (ATCS)} speitude cest bartery was investigated within

the framework of the Uniform Guidelines on Employee Selection Procedures (29 CFR 1607). First, the adverse impact

of using a composite score on the test battery was evaluated oc an archival sample of 170,578 job applicants. Mean score |

differences by gender of 0.35 SD in the favor of men were found, suggesting char adverse impact on women could be :

expected from use of test scores in selection. Analysis of selection: rates by gender found that women were dlassified as |

clighle for employment consideration on the basis of composite tex: scores at 2 lower rave (38.5%) chan men (50.4%).
hierarchical regression analysis (Lautenschiager & Mendoza, 1986) wes used ro investigate differential

prediction of performance in initial ATCS waining at the Federal Aviation Administration Academy in a sampic of 9,552

first-time compecicive entrances. Analysis based on correlations osrrected fer explicit und implicic restriction in range

found significant differences in slopes and intercepts by gender, suggesting that separate regression equations were

appropriate to peedict Acederny performance for the genders. The practical significance of the statistically small effect

§ sizes is considered within the frarnewrork of current equal employment oppormunity legislation and case law im che

discussion of these results. The altersative explanation that these resulss reflect gender differences in job-relaced abilities,

zathez chan test bias, is considered.

sawn reales amannsnnenamnce-evenmorascie te seecen nme

V7. key Weeds
ALCS celeczion; Test bias; Hierarchical regression;
Adverse impect, Fairness; Equal employment opportunity;

neon prone

18. Dettinution Sotement
Document is evailable zo che public through the
National Technical Information Service,

Jeenmceneninn nen ridin er cmniescmnnencitaensne sree am

Personne! selection SpringSeld, VA 22161
19. Securty Clase. (of ris report) | 20 Secinty Gam (PE page) i 22. No of Pages : 22. Poe
| Unclassified Unclassified I 21 i

fom DOt ® $700.7 6-72) feagrocuction of competnd page aufhowed

i


Teble

Sampie and corrected correlation matrix for -Fferentia!

prediction analysis by gender

TTMc SEX

TMC_SEX SCREEN

TMC 8.1479"
SEX 0.08186

TMC_SEX 9.0296"  0.9986°°°
SCREEN 0.1844") -0.0877°"

O06 9.4724
C738" — -O.8661
0.0408

-2.0847°°°

Aiuto” As described ia Tablt 5, cocrelstions betecet:
‘SCREEN and TMC, SEX, and TMIC_SEX aa the
‘ager cighthand corner are comvected. asd therefore.
Ro signifcamer tests ae vepceted

"p< Ol, “rs.

Figure 3
Grterion (SCREEN score dstibuton by gender i research sampie (6 =
2400 Males Femaies

2003

1600



“eyed rom woppaastias omnyoemsagy UAdp-dads JO

ny Fd, rooy pus 3(Bo] 295 | ssn Bk 80g ‘SjoporT Pompoadeross ‘A FRY Jo UueprEOD WIL paysURLIO) syuhpry y8sI0N
XAS + OWL ~

VIN OWL ghnosp ‘a Heeq
cco VOLED — gag LW BEE 1s100° LEO XASOWL + OWL
coo VE'STON 6100 XAG OVEL+ MAS + OW. = slacsonuy sonposdssos:) “A [1M
ae0BP LED — goo 'SEZ 18100" ZELTO XS + OWL
soo SIOL ovo XS” OWL + XS + OWL sadojg sdnoaty “A (1M
cc VESIOL ose DLLIE £8100 6170 XAS OWL + Xas + OWL
wont ¥ EVLE ZErz'0 OWL SWIG {IWIFAG [If “A DEG
~~ a ~ av av a (pow siskspony

eBuLs Uy UORZ UGE! 40) Ps}591200 KUNE UOHBIOLOD

Jo sys8q Lo sepund hq oydeues Yorooaad Uy eBIq 160) 40} ejehjuuD UOjesoIHOI (ROH /E/OR uop-dere Jo mynseY

BOREL