Predicted SCREEN

40 4
30 4..
QO fon eeseeccseesececeeseecsueeneeesneesssansseneessnaeeecesseereeed = White
MO fo seese ese eeeeececceeeece cece che eeeeescnnsimeeceneseineneeseecensel = African-American] ||
0 1 T T T T
70 75 80 85 90 95 100
TMC

Figure 7: Regression of SCREEN on TMC for African Americans and whites, with corrections for restriction

in range

Test bias

Our step-down hierarchical regression analyses
found that the written ATCS aptitude test battery
exhibited predictive bias, as defined by the Uniform
Guidelines at 29 CFR 1607.14.B.(8).(a) and Cleary
(1968). That is, the white regression line over-
predicted the performance of the protected group.
Moreover, the white regression line was indistin-
guishable from the combined sample regression.
Therefore, we concluded that the common regression
line, used as the operational basis for determining test
weights and cut score (Rock, Dailey, Ozur, Boone, &
Pickrel, 1982), over-predicted the performance of
African Americans in the FAA Academy ATCS Non-
radar Screen. Adverse impact and over-prediction of
training and job performance has been reported for
tests of cognitive ability similar to the ATCS test

17

battery such as the General Aptitude Test Battery
(GATB) (Hartigan & Wigdor, 1989; Hunter, 1983;
Schmidt, 1988; Wigdor & Garner, 1982; Wigdor &
Sackett, 1993) and the Armed Services Vocational
Aptitude Battery (ASVAB; Dunbar & Novick, 1988).
These outcomes do not occur because scores on tests
such as the ATCS aptitude test battery mean some-
thing different for African Americans. The tests are
not biased in that sense. Rather, the different pass
rates and over-prediction of subsequent performance
in initial training results from the interplay of two
factors, as has been found in other selection settings
(see Gordon, Lewis, & Quigley, 1988): the lower
average scores for African Americans relative to whites,
and the less than perfect validity of the test scores.



100° 5 4,4

#0 0S T1- ELel OV9L PS'8 66LT 6559 787 92098 % 389) S]ID{S JO]]ONUOD Ls

ane UT Esl 1€'P9 tpS's SOOT 89'rS = 787 ‘auo9s as}or0x0 AroyBIOQUT PIPBIS 9 JOC 3S9q OBLIAY = SDAVEVT

«4A TTS" Ore £r'06 = ZS‘ £$6 LOL8 78% loos % 389) aseyd oatsuoyoidulos ldo

#4486°01- 489 = OP'E6ZHS‘B vel O88 = 787 91098 % OB¥IOAL 389} YOOIg MUSPEIY OAVIDOTA

4 as Ww N as Ww N uondusseq, == uauodutod,
SuU AN UBOHOUYy UBOLyY

aoe Aq S01008 jusuodui0a UOVeWO N33YWOS
S$ oqeL

i



American Psychological Association, & National
Council on Measurement in Educaiion, 1985). One
might suspect, for example, that education and scores
on the written aptitude test might be confounded in
view of the generally positive correlation between
such tests and educational attainment: The group
with lower scores on an aptitude test battery might
have lower overall educational levels than the other
group with higher scores. Overall, African Americans
in the research sample had slightly lower educational
levels than whites: 22.7% of African Americans re-
ported completing a baccalaureate degree, compared
~ with 32.2% of whites (Z = -3.37, p < .001). The
correlation between education and TMC was .089 (p
<-01). While low, this correlation is about the same
magnitude as the uncorrected correlations between
TMC and SCREEN reported in Table 6. Thus, a
plausible alcernative explanation for our resules might
be the diffesing educational levels for the groups.
Another unmeasured variable that might influence
our results might be age at entry into the FAA
Academy. Age has been found to be related to Screen
outcomes in previous research (Collins, Nye, &
Manning, 1990; VanDeventer, 1983).
Introduction of these additional variables, and
their multiple interactions with race, aptitude, and
each other, however, presents a difficult analytic
ptoblem. Procedures for conducting a hierarchical,
step-down regression analysis with multiple vari-
ables, and interpretation of the results from such an
analysis, have not been defined. However, statistical
procedures for testing the fit of a measurement and
structural model for different groups or populations
are relatively well established. An alternative analytic
approach would be to conduct a structural equations
analysis of the relationships between aptitude (TMC),
education, age 2t entry, and performance in the FAA
Academy Screen. For example, the model! could be
developed on the basis of half of the white sample,
and cross-validated on the other half. The degree thar
the cross-validated white model fit the data for Afri-
can Americans could then be formaily tested. Such an
approach would also allow a very focused test of the
equality of the TMC - SCREEN path parameter for
African Americans and whites. Moreover, such a
strong analytic method would allow for the more
precise specification of method, predictor, and crite-
rion measures and latent constructs required to assess
ctiterion as well as predictor bias in criterion-relared
validation (Schmitt, Pulakos, Nason, 8& Whitney, 1996).

21

CONCLUSION

In conclusion, on the one hand, the ATCS
written aptitude test battery in use by the FAA
through February 1992 may have operated as an
“engine of exclusion” (Seymour, 1988) in terms of
differential selection rates. Historical data suggest
that the test battery may have excluded more
African Americans than whites; however, a defini-
tive selection rate analysis was nor technically
feasible in this study due to the lack of racial
identifiers for applicants. On the other hand, the
analyses indicated that the written ATCS aptitude
test battery, in accordance with the definition
used by Cleary (1968), the Uniform Guidelines,
and relevant professional testing standards and
principles, appears to have been biased in that the
performance of African Americans was over-
predicted by the common (white) regression of
aptitude scores on subsequent performance in
training. That is, use of the recommended cut-off
of 90 on the composite predictor score TMC
would have resulted in a greater incorrect accep-
tance (false positive) rate for African Americans
than whites. These results appear to be consistent
with the general findings that ability tests do not
under-predict the performance of minorities (Linn,
1994; Schmidt, 1988). We recommend three ad-
ditional research studies as next steps. First, we
recommend a careful and detailed assessment of
how, and to what degree, inappropriate systematic
variance might have been introduced into the 13
scores representing the degree to which students
mastered required air traffic control skills and
knowledges in the FAA Academy screening pro-
gtam. The second step is to incorporate those
findings on criterion bias with the predictor and
other exogenous measures such as education and
age into a structural equations model as the basis
for testing hypotheses about differential predic-
tion in the selection of controllers. The final step
we recommend is to examine the practical and
organizational consequences of operational use of
the majority (white) regression line as the basis for
historical selection decisions, in view of the appar-
ent over-prediction of the performance of African
Americans in the FAA Academy Screen.



1607.14.B.(8).(b)) and Standards for Educationaland
Psychological Testing (Standards 1.20, 1.21, 1.22, p.
17) was warranted. Our formal null hypothesis was
no difference in the predictive validity of the test

battery by race.
METHOD

Sample

Archival test data were available for a sample of
170,578 (42.2%) persons from the 403,997 job ap-
plicants who took the written ATCS aptitude test
battery from 1981 through 1992. Between October
1985 and January 1992, 14,392 persons entered as
students into initial controller training at the FAA
Academy. Most (79.2%) of these new hires had
competed under civil service regulations and were
entering the Academy for the first time. The differen-
tial prediction analysis was based on the 8,824 stu-
dents with full and complete racial identification,
predictor, and criterion data. There were 8,542 (96.8%)
whites and 282 (3.2%) African Americans in the re-
search sample. Demographic information for the appli-
cantsample, all 1986-1992 FAA Academy entrants, and
the research sample are presented in Table 1.

Measures

Predictors. The written ATCS aptitude test bat-
tery was the initial hurdle in the ATCS selection
process, and consisted of three tests: (a) the Multiplex
Controller Aptitude Test (MCAT); (b) the Abstract
Reasoning Test (ABSR); and (c) the Occupational
Knowledge Test (OKT). The MCAT was a timed
paper-and-pencil civil service test (OPM test No.
510) simulating activities required for control of air
traffic. Aircraft locations and direction of flight were
indicated with graphic symbols on a simplified, simu-
lated radar display (Figure 1). An accompanying
table provided relevant information required to an-
swer the item, including aircraft altitudes, speeds,
and planned routes of flight. MCAT test items re-
quired examinees to identify situations resulting in
conflicts between aircraft, to solve time, speed, and
distance problems and also to interpret the tabular
and graphical information. The ABSR was a 50-item
civil service examination (OPM test No. 157). To
solve an item, examinees determined what relation-
ships existed within sets of symbols or letters. The
examinee then identified the next symbol or lecter in

the progression, or the element missing from the set.
A sample ABSR item is presented in Figure 2. The
OKT was an 80-icem job knowledge test that con-
tained items related to seven knowledge domains
relevant to aviation generally, and to air traffic con-
trol phraseology and procedures, specifically. The
OKT was developed as an alternative to self-reports
of aviation and air traffic control experience. The
OKT was found to be more predictive of perfor-
mance in ATCS training than self-reports (Dailey 8&
Pickrel, 1984; Lewis, 1978).

The development of the written ATCS aptitude
test battery has been extensively described elsewhere
(Brokaw, 1984; Collins, Boone, & VanDeventer,
1984; Manning, 1991; Sells, Dailey, & Pickrel, 1984).
The test-retest correlation for the MCAT was esti-
mated at .60 in a sample of 617 newly-hired control-
lers (Rock, Dailey, Ozur, Boone, & Pickrel, 1982, p.
59). Parallel form reliability, as computed on the
same sample, ranged from .42 to .89 for various
combinations of items (Rockeral., p. 103). Lilienthal
and Pettyjohn (1981) examined internal consistency
and item difficulties for ten versions of the MCAT.
Cronbach’s alpha for the ten versions ranged from
-63 to .93; the alphas for 7 of the 10 versions were
greater than .80. The available data suggest that the
MCAT had acceptable reliability. In contrast, no
item analyses, parallel form, test-retest, or internal
consistency estimates of the ABSR test have been
reported.

Scoring of the test battery was done initially by
summing the MCAT and ABSR scores, as shown in
Table 2. The resulting total weighted score (TWS)
was then transformed toa score with a mean of 70 and
maximum of 100, known as the Transmuted Com-
posite Score (TMC). About half ofall applicants were
expected to score at or above the mean (Rock, Dailey,
Ozur, Boone, & Pickrel, 1984b). Applicants with
three years of general work experience, four years of
college, or any combination of education and experi-
ence equivalent to three years of general experience,
needed a TMC score of 75.1 to be considered for
employment. Applicants with one year of graduate
study, superior academic achievement, or specialized
aviation or air traffic control experience, required a
TMC score of 70 to qualify for employment consid-
eration (Aul, 1991). Applicants not meeting these
criteria were ineligible for consideration for employ-
ment as controllers.


Impact of over-prediction

The impact of this apparent over-prediction on
African Americans is illustrated by an analysis of the
selection decisions that would have been made using
a strict cut-off on TMC as the hiring criterion. In
such an analysis, hire/no hire recommendations are
crosstabulated with job outcomes to create a decision
table. As previously noted, a TMC score of at least 90
was required to predict a passing score of 70 in che
EAA Academy Screen, based on the common (white)
regression line. Based on that statistical relationship,
an operational decision rule of hiring applicants
scoring at 90 or above was generally used by the FAA.
The policy of preferring applicants with scores of 90
or above was made explicit in 1990 (Associate Ad-
ministrator for Human Resources Management,
1990). TMC was therefore recoded into a dichoto-
mous variable (0 = TMC < 90; 1 = TMC 2 90) to
represent this operational hiring criterion. The di-
chotomized variable was then crosstabulated with
FAA Academy ATCS Non-radar Screen pass/fail
outcomes by race, as shown in Table 9.

Arrangement of the crosstabulation

By convention, two-by-two predictive tables such
as Table 9 are arranged such that the rows are defined
by test predictions (positive on rop, negative on the
bottom), and columns are defined by criterion results
(negative on the left, positive on the right)(Gordon,
Lewis, & Quigley, 1988). The top row for each
group, therefore, consists of two cells: false positives
(in selection, also known as incorrect acceptances),
and cue positives Gaceect accepances}. The bottom
row for each group consists of rwo cells: true nega-
tives (correct rejections), and false negatives (also
known as incorrect rejections). False positives are
those persons who had a TMC of 90 or greater (TMC
> 90) and who subsequently failed the Screen
(SCREEN < 70). True positives are those cases with
a TMC score of 90 or greater and who subsequently
passed the Screen (SCREEN 2 70). Troe negatives
are those individuals who had a TMC of less than 90
and also failed the Screen (SCREEN < 70). Finally,
false negatives are those persons with TMC scores of
Jess chan 90 and who passed the Screen (SCREEN > 70).

Analysis of crosstabulation cells

Both temporal ordering and direction determine
which varidphe snoukd be used as the base for calculation
of the percentages showing the effect in acrosstabulation

18

(Davis, 1971; Zeisel, 1957). In this case, che written
ATCS test battery preceded the Screen, often by several
months (Aul, 1991). The putative causal role belongs to
the predictor test; the analytic goal is to estimate perfor-
mance on the criterion (SCREEN) from the predictor
score (TMC). Therefore, the percentages of true and
false positives and true and false negatives in a fourfold
classification table such as Table 9 should be calculated
horizontally, using the selection test recommendation
(hire, no hire) as the base for such calculations (Gordon,
Lewis, & Quigley). The focus ofa fourfold classification
table is the accuracy of predictions made on the basis of
test scores about future job performance as represented
by the decision errors. Decision errors provide the data
for evaluating the impact of over- and under-prediction
of subsequent training or job performance (Gordon,
Lewis, & Quigley).

Decision errors: Incorrect rejections (false nega-
tives). The incorrect rejection rate for each group is
defined as the ratio of (2) the number of persons with
TMC scores of less than 90 who passed the Screen
(the lower right cell for each group in Table 9) to (b)
the marginal row total for persons with TMC scores
of less than 90 for each group. The actual job perfor-
mance for persons in this cell for each group was
under-predicted by their predictor test score. There
were 39 African Americans with TMC scofes of less
than 90 who passed the FAA Academy, out of 128
total with TMC scores of less than 90, for an incor-
rect rejection (false negative) rate of 30.5%. As shown
in Table 9, there were 1,684 whites with TMC scores
of less than 90 who passed the Screen our of 3,046
whites with TMC scores of less than 90, for an
incorrect rejection rate of 55.3% (1,684/3,046). If
the “TMC > 90” hiring rule had been used, the
proportion of rejections that would have been incor-
rect for African Americans was significantly less than
the proportion for whites (Z = -5.52, p < .001).

Decision errors: Incorrect acceptances (false posi-
tives). The incorrect acceptance or false positive rate
for each group is defined explicitly as the ratio of (a)
the number of persons with TMC scores of 90 or
greater who failed the Screen (b) to the marginal row
total of persons with TMC scores of 90 oF greater.
There were 84 African Americans who failed the
Screen out of a total of 154 with TMC scores greater
than or equal to 90, for an incorrect acceprance rate
of 54.5%. Of the 5,496 whites with TMC scores of
90 ox greater, 1,768 failed the Screen, for an incorrect
acceptance rate of 32.2%. If the “TMC > 90” hiring


Manning, C.A., Della Rocco, P.S., & Bryant, K. (1989).
Prediction of success in air traffic control field train-
ing as a function of selection and screening perfor-
mance. (DOT/FAA/AM-89/6). Washington, DC:
Federal Aviation Administration Office of Avia~
tion Medicine. (NTIS No. ADA209327)

Millsap, R.E. (1988). Sampling variance in the correla-
tion coefficient under range restriction: A monte
carlo study. Journal of Applied Psychology, 74,
456-461.

Pedhazur, E.J. (1982). Multiple regression in behavioral
research (2nd Ed.). New York: CBS College Pub-
lishing. ’

Rock, D.B., Dailey, J.T., Ozur, H., Boone, J.O., &
Pickrel, E.W. (1982). Selection of applicants for the
air traffic controller occupation. (DOTIFAA/AM-
82/11). Washington, DC: Federal Aviation Ad-
ministration Office of Aviation Medicine. (NTIS
No. ADA122795/8)

Rock, D.B., Dailey, J.T., Ozur, H., Boone, J.O., &
Pickrel, E.W. (1984a). Conformity of the new
experimental test battery to the Uniform Guide-
lines on Employee Selection Requirements. In
S.B. Sells, J.T. Dailey, & E.W. Pickrel (Eds.),
Selection of air traffic controllers (pp. 503-542).
(DOT/FAA/AM-84/2). Washington, DC: Fed-
eral Aviation Administration Office of Aviation
Medicine. (NTIS No. ADA147765)

Rock, D.B., Dailey, J.T., Ozur, H., Boone, J.O., &
Pickrel, E.W. (1984b). Validity and utility of the
ATC experimental tests battery. Study of Acad-
emy trainees, 1978. In S.B. Sells, J.T. Dailey, &
E.W. Pickrel (Eds.), Selection of air traffic control-
lers(pp. 459-502). (DOT/FAA/AM-84/2). Wash-
ington, DC: Federal Aviation Administration
Office of Aviation Medicine. (NTIS No.
ADA147765)

Sackett, P.R. (1996). Interpreting the ban on minority
group score adjustment in preemployment test-
ing. In R.S. Barrett (Ed.), Fair employment strate-
gies in human resource management (pp. 246-256).
Westport, CT: Quorum Books.

Sackett, PR, & Wilk, S.L. (1994). Within-group
norming and other forms of score adjustment in
preemployment testing. American Psychologist, 49,
929-954.

24

Schmidt, F.L. (1988). The problem of group differ-
ences in ability test scores in employment selec-
tion. Journal of Vocational Behavior, 33, 272-292.

Schmitt, N., Pulakos, E.D., Nason, E., & Whitney,
DJ. (1996). Likability and similarity as potential
sources of predictor-related criterion bias in vali-
dation research. Organizational Behavior and
Human Decision Processes, 68, 272-286.

Schroeder, D.J., Broach, D., & Young, W.C. (1993).
Contributions of personality to the prediction of
success in initial air traffic control specialist training.
(DOT/FAA/AM-93/4). Washington, DC: Fed-
eral Aviation Administration Office of Aviation
Medicine. (NTIS No. ADA264699)

Sells, S.B., Dailey, J.T., & Pickrel, E.W. (Eds.) (1984),
Selection of air traffic controllers. (DOT/FAA/AM-
84/2). Washington, DC: Federal Aviation Ad-

* ministration Office of Aviation Medicine. (NTIS
No. ADA147765)

Seymour, R.T. (1988). Why plaintiffs’ counsel chal-
lenge tests, and how they can successfully chal-
lenge the theory of “validity generalization.” Jour-
nal of Vocational Behavior, 33, 331-364.

Siegel, A.I. (1978). Miniature job training and evalua-
tion as a selection/classification device. Human
Factors, 20, 189-200.

Siegel, A.I. (1983). The miniature job training and
evaluation approach: Additional findings. Person-
nel Psychology, 36, 41-56.

Siegel, A.L, &¢ Bergman, B.A. (1975). A job learning
approach to performance prediction. Personnel
Psychology, 28, 325-339.

Society for Industrial and Organizational Psychology.
(1987). Principles for the validation and use of
employee selection procedures. (3" ed.). College
Park, MD: Author.

SPSS, Inc. (1989). SPSS-X User's Guide. Chicago, IL:
Author.

U.S. Congress. (January 20, 1976). House Committee
on Government Operations: Recommendations on
air traffic control training. Washington, DC: Gov-
ernment Printing Office.



tule had been used, the proportion of “recommended
hire” decisions that would have been incorrect (false
positives) for African Americans was significantly
greater than the proportion for whites (Z = 5.81, p<
-001). In other words, it is likely that significantly
more African-American than white candidates would
have been incorrectly accepted on the basis of their
test scores using a TMC score of 90 or greater as the
hiring rule.

Interpretation of decision errors. Use of the “TMC
290” rule would have resulted in significantly differ-
ent proportions of decision errors for whites and
African Americans. Specifically, significantly more
of the African Americans with aptitude scores greater
than 90 wenton to fail training than would have been
expected on the basis of the common (white) regres-
sion line. That is, the performance of African Ameri-
cans in the Screen was over-predicted by TMC.
Moreover, the burden of incorrect rejections (false
negatives) on the basis of ATCS aptitude test scores
did not fall disproportionately on minority candi-
dates, in contrast to results from previous research on
the GATB (Hartigan & Wigdor, 1989; Wigdor &
Sackett, 1993).

Interpretive issues

Three issues might be raised as objections or con-
cerns to our interpretation of the results. First, the
statistical effects detected in the differential predic-
tion analyses were generally small and detectable only
with very large samples. Those small effects were
more pronounced with corrections for restriction in
range. One could argue, therefore, that the results
were artifactual (cf. Hunter, Schmidt, & Rausch-
enberger, 1983), and had little practical significance.
‘We would counter by noting that the FAA controller
selection process was a large scale selection system.
Small effect sizes have significant practical effects in
large-scale selection systems such as that for control-
lers (Schroeder, Broach, & Young, 1993). Corrected
correlations may also provide more accurate esti-
mates of test validity, particularly in large samples
and under stringent selection ratios (Bobko, 1983;
Millsap, 1988), such as encountered by the FAA.
Uncorrected coefficients appear to be downwardly
biased estimates of the true population validity coef-
ficients (Lee, Miller, 8¢ Graham, 1982). Therefore,
differential prediction analyses based on corrected
correlations provide less biased estimates of popula-
tion effects. Moreover, study factors such as disparate

20

sample sizes between groups and the small moderat-
ing effect may have reduced the overall statistical
power of the analysis (Aguinis 8 Stone-Romero,
1997). Yer statistically a significant moderator effect
for race on the validity of the written ATCS aptitude
test battery was detected. Finally, we believe that
these effects cannot be lightly dismissed, in view of
the very real practical consequence for the FAAATCS
selection program: a higher proportion of African
Americans failed than would have been expected on
the basis of aptitude test scores.

Second, one might argue that the observed differ-
ential prediction of SCREEN on the basis of TMC
for African Americans and whites might be attribut-
able to bias in the criterion measure. The mean
criterion scores for the groups were significandy
different (Table 3), and the proportion of African
Americans passing the Screen (38.7%) was signifi-
cantly lower than the white proportion (63.4%; Z=
-8.34, p < .001). The pass rate ratio at the Screen was
-61, indicating that African American trainees passed
the Screen at 61% of the white pass rate. Yet a
difference in average criterion scores between groups
does not establish bias; “... the presence or absence of
bias cannot be detected from knowledge of criterion
scores alone” (SIOP, 1987, p. 10). Bias, in this sense,
is the extent to which a criterion includes unwanted
systematic variance. Unwanted systematic variance
might be introduced into supervisory ratings, for
example, by rater errors such as halo, leniency, stereo-
typing, and ratee-by-rater race interactions (Ford,
Kraiger, & Schechtman, 1986; Kraiger & Ford, 1985).
But, as noted by Ford and his co-authors in their
review of the literature, the effects found for rating
errors and rater-by-ratee interactions do not account
for all of the variability between groups, and do not
preclude the possibility that actual performance dif-
ferences between groups exist. The degree to which
unwanted systematic variance might have been intro-
duced into the measures comprising the SCREEN
composite criterion score has not been formally as-
sessed; therefore, the possibility of criterion bias
cannot be dismissed. Further investigations of the
process by which the criterion measures were gener-
ated and the degree to which possible biasing factors
account for group differences are required.

Third, unmeasured variables may have been con-
founded with the predictor, resulting in a defective
study design (Anastasi, 1988; see Standard 1.22, p.
17, American Educational Research Association,



With corrections for restriction in range

However, as shown in Table 3, predictor scores
were severely restricted. Consequently, evidenc based
on those uncorrected correlations may be somewhat
misleading as to the predictive bias of the predictor
(29 CFR 1607.14.B.8.(c)). Analyses based on corre-
lations corrected for explicit and implicit restriction
in range may provide a better assessment of the
predictive bias of the OPM test battery with respect
to the applicant population. Corrected correlations
are presented in the upper right-hand triangle of the
matrix in Table 6. The estimated population correla-
tion between TMC and performance in the Academy
Screen increased from .199 to .519 with correction
for explicit restriction in range. After correcting for
incidental restriction in range, the correlation
between RACE and SCREEN increased to -.208,
as did the correlation bewween the cross-product
(TMC_SCREEN) and SCREEN (-.191).

The results of the differential prediction analysis,
using the step-down hierarchical regression analysis
based on the corrected correlations, are presented in
Table 8, The null hypothesis of a common regression
line was rejected in the basic versus full model com-
parison, suggesting the presence of some degree of
test bias. The increment in Rt associated with the full
model over the basic model was significant (AR =
056, AF = 366.54, p S001). Next, the null hypoth-
esis of same slopes by race was rejected in the full
versus subgroup model analysis, The subgroup model
(predictor and group membership) explained less
variance than the full model (AR’ = -.026, AF =
345.85, p S .001). Following the logic illustrated in
Figure 4, the full and cross-product models were next
compared to determine if the intercepts were differ-
ent for African American and white applicants. The
null hypothesis of same intercepts was also rejected,
with removal of RACE leading to a significant reduc-
tion in explained variance (AR* = -.030, AF = 391.29,
PS .001). Overall, the results obtained with the
corrected correlations indicated the need for separate
regression equations for African American and white
applicants, with different slopes and intercepts for
thetwo groups. Therefore, correlations between TMC
and SCREEN were computed for African Americans
and whites separately, based on the research sample
data. These correlations were then corrected for

explicit restriction in range based on the applicant
sample TMC SD. The corrected correlations were
then submitted to regression analysis. The resulting
equation for whites was

PREDICTED SCREEN =-37.85 + (1.200"TMC), (3)

compared with an equation for African Americans of

PREDICTED SCREEN = 7.63 + (0.623*TMC. (4)

These regression equations are plotted in Figure 7.
This analysis based on corrected correlations leads to
the same conclusion as the preceding analysis: TMC
appeared to be biased using Cleary’s (1968) defini-
tion of test bias. The performance of African Ameri-
cans in the FAA Academy Non-radar ATCS Screen
would be over-predicted from the white regression
line. Given that the combined sample regression line
is essentially identical to the white regression line, it
is fair to conclude that the performance of African
Americans in the Screen would be over-predicted by
the combined sample regression line.

DISCUSSION

Pass rate differences

Evaluation of a selection test under the Uniform
Guidelines includes (a) an assessment of differences in
pass rates between groups arising from use of the test,
and if differential pass rates are demonstrated, (b) an
investigation of predictive bias associated with the
test. Our study is silent as to any differences in pass
rates for whites and African Americans arising from
use of the written ATCS aptitude test battery, due to
the lack of racial identifiers for applicants. However,
Previous research indicated the African American
pass rate on the ATCS aptitude test battery was likely
to differ significantly from that of whites. Previous
research also concluded that the battery would likely
exhibit differential prediction for African Americans
(Rock, Dailey, Ozur, Boone, & Pickrel, 1982, p.
153). Therefore, an evaluation of the predictive bias
for the battery was conducted, as required by the
Uniform Guidelines and relevant professional selec-
tion testing standards and principles.



Table 4
Mean predictor (TMC) and criterion (SCREEN) score differences by race
Variable Race N M SD SE t df
TMC White 8,542 91.67 477 0.052 6.56°** 8,822
African 282 89.77 5.53 0.329
American
SCREEN White 8,542 FAL ANE) 9.128 AL 6Ate* 8822
African 282 63.57 13.73 0.818
American
“) <.001
1800 60
—o White
~#- African-American
C711) | fe CepeeeeseeesseseestesestsessesessScett CDi CSCnCSEcEenCeCeSnESEnES | ‘osUseCESEEESSESISEEESSS +50
1200 os epee nnnnen nr rer errr rerert lore rerrrreny | Conerien/ sore itnnnn maroon ios seasceeeeseoat f 40
=
z
GOO onan ere nnectccecccceeetttttttttte fice eect dN Circe + 30
=
£
=
C11 eee eee ee cece p nD DODO ESD E ED EDESESESOEEIEEE | SESSSEESI SUEDE SE SEEORESORECESD SELEEDONCESES + 20

70 72 74 76 78 80 82 84
TMC

Figure 3: Predictor (TMC) score distribution by race

T

86 88 90 92 94 96:98 100

—+0

African-Americans (N)


AIRCRAFT ALTITUDE SPEED ROUTE ¢
10 7000 480 = AGKHC w —€ A
70

20 7000 480 BGJE
30 7000 240 AGJE
40 6500 240 CHKJF
50 6500 240 DIKGB
60 8000 480 DIKJE
70 8000 480 ©FJKID

SAMPLE QUESTION

WHICH AIRCRAFT WILL CONFLICT?

A. 60 AND 70

B. 40 AND 70

C. 20 AND 30

D. NONE OF THESE

04812
nas
MILEAGE SCALE

Figure 1: Example Multiplex Controller Aptitude Test (MCAT) item

Symbols .
a de ?] OAGGal

A BC DE

LINIVA?T NRL SY
Aspc¢cODE

2.

Letters
‘1) XCXDXEX A)FX 3B)FG c)X¥ D) EF £&)XG

2) ARCSETG = 4)HI_ 8) HUG) UJ Dy UL IV

Figure 2: Example Abstract Reasoning Test (ABSR) item

4

a


Table 1
Demographic characteristics for applicant sample, all 1985-1992 FAA
Academy entrants, and the research sample

FAA Academy entrants

Applicant FAA Academy Research
Characteristic sample* entrants sample
(N = 170,578) (N=14,392) (N=8,824)

Race
White 12,366 8,542
African American 819 282
Other? ; 811
Missing 396

Education
<High School 404
High School 28,147 1,576 969
Some college 82,414 7,150 4.928
Bachelor's degree 54,583 4,745 2,818
Advanced degree 3,934 176 109
Missing 1,096 145

Age
Mean 26.21 25.78
sD 4.90 2.86

Notes: *Racial identification and age data not available for applicant sample.
‘Other includes American Indian/Alaskan Native, Asian/Pacific Islander, and
Hispanic.



‘sisAyeue UojssouBox [eoryosesory

100 >4,,4 umop-deis Jo Moly Pue a(Bo} 105 | aun 29g “sjepou onposd-sso1o “A JJry Jo VoswEdwoD yu payouTIUsD SISKIeLY —-sOION
dnosaqng ‘a oiseg
44086 6681 wan 60 16E 0t0'o- vrS'0 POVYsOWL + OWL
aaa STP VES BOVaAOWL + SOVE + OWL sidaouayuy onposd-ssos3 “A ]]N]
ene00'T881 auS8'SHE 970°0- Lys'0 SOVa+ OWL
aneCl SIDI oLs‘0 SOVEKOWLL + BOVE + OWL sadojg :dnoaSqng *a yny
weeC’ SIVT een SIDE 9500 OLS‘0 SOVas+OWL + FOVA + OWL
wen DE CSTE 6Is‘0 OWL S¥Iq [[8I2AQ *[[N,J “A oIseg
d dv av a Tepow siskyeuy

eBues uy uoHo1yS21 105 paye1109 xyJEU LOHEIEI0D JO S}seq UO eDeJ Aq SeIq 180} 10} sishjeUR UO|ssalBe! eolyoLEJEIY UMOP-de}s jo SINsoL

8 o1q0)

16



wrerdoid ws0i0g sepes-1ON SOLY Awopeoy WV UI 23005 syoduioD [eu = NAMES ‘HOV PUP WL
30 0nposd-ss019 = FOV A OWL “(unoyiamy uvoufy = | ‘anya = 0) snyers Aoun = AVY ‘Asoneq 1801 SLY YoNUM wWosy s100g asoduI0 PoynusUBLL, = DAL,
“ojdures yueoyjdde 103 21qQuoyjdde j0u as008 waalag ‘oduses yueoydde 103 a]qoyteae jou Byepaoey, Sato

6516 OTLZ «BTIT  SBIL Lp66 OVLZ «OST «= 97ZL NagYOS

OOOO! OOO zest LBZ OO'00T «000 = OSE BT OT BOVUOWL

or oO 810 00 ool oO oro ozo aova

OOOO! 900L Ish 1916 OOOO] OOOL «HS 30°16 OOOor ES6l LEP oes OWL

xeW WA as WwW xe wW as WwW xe uw as Ww eigen,
(28'8=N) ofdutes yoressoy (Z6€‘PI=N) stuenua Awapeoy Wy <(8LS‘OLI=N) o[dures yueoyjddy

ejdwes yorweses ey; pu ‘sjueque Awepeoy yy4 2661-SR6} Ile ‘siueoydde gof Jo ejdwes souelejes 10) Sonsneys eayduoseg
€ equ,


REFERENCES

Aguinis, H., & Stone-Romero, E.F. (1997). Method-
ological artifacts in moderated multiple regres-
sion and their effects on statistical power. Journal
of Applied Psychology, 82, 192 - 206.

American Educational Research Association, American
Psychological Association, & National Council
on Measurement in Education. (1985). Standards
for educational and psychological resting. (Sth ed.).
‘Washington, DC: American Psychological Asso-
ciation.

Anastasi, A. (1988). Psychological testing (6% ed.). New
York: Macmillan.

Associate Administrator for Human Resources Man-
agement. (1990, August). Report on the FAA em-
ployment manager's conference, June 25-29, 1990,
Arlington, VA. Washington, DC: Federal Avia-
tion Administration Staffing Policy Division.

Aul, J.C. (1991). Employing air traffic controllers. In
H. Wing & C.A. Manning (Eds.), Selection of air
traffic controllers: Complexity, requirements, and
public interest (pp. 7-12). (DOT/EAA/AM-91/
9). Washington, DC: Federal Aviation Adminis-
tration Office of Aviation Medicine. (NTIS No.
ADA238267)

Bardert, C.J., Bobko, P., Mosier, S.B., & Hannan, R.
(1978). Testing for fairness with a moderated
multiple regression strategy: An alternative to
differential analysis. Personnel Psychology, 31,233-
241.

Bobko, P. (1983). An analysis of correlations corrected
for attenuation and range restriction. Journal of
Applied Psychology, 68, 584-589.

Broach, D. (1998). Air traffic control specialist apti-
tude testing, 1981-1992. In D.Broach (Ed.), Re-
covery of the FAA air traffic control specialist work-
force, 1981-1992 (pp. 7-16). Washington, DC:
Federal Aviation Administration Office of Avia-
tion Medicine.

Broach, D., & Manning, C.A. (1994). Validity ofthe air
traffic control specialist non-radiar screen as a predic-
tor of performance in radar-based air traffic control
training. (DOT/FAA/AM-94/9). Washington,
DC: Federal Aviation Administration Office of
Aviation Medicine. (NTIS No. ADA279745)

22

Brokaw, L.D. (1984). Early research on controller se-
lection. In S.B. Sells, J.T. Dailey, & E.W. Pickrel
(Eds.), Selection of air traffic controllers (pp. 39-
78). (DOT/FAA/AM-84/2). Washington, DC:
Federal Aviation Administration Office of Avia-
tion Medicine. (NTIS No. ADA147765)

Cleary, T.A. (1968). Test bias: Prediction of grades of
negro and white students in integrated colleges.
Journal of Educational Measurement, 5, 115-124.

Cohen, J. (1988). Statistical power analysis for the behav-
ior sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Cohen, J., & Cohen, P. (1975). Applied multiple regres-
sionlcorrelation analysis for the behavioral sciences,
Hillsdale, NJ: Lawrence Erlbaum Associates.

Collins, W.E., Boone, J.O., & VanDeventer, A.D.
(1984). The selection of air traffic control special-
jst: Contributions by the Civil Aeromedical Insti-
tute. In S.B. Sells, J.T. Dailey, & E,W. Pickrel
(Eds.), Selection of air traffic controllers (pp. 79-
112). (DOT/FAA/AM-84/2). Washington, DC:
Federal Aviation Administration Office of Avia-
tion Medicine. (NTIS No. ADA147765)

Collins, W.E., Nye, L.G., & Manning, C.A. (1990).
Studies of poststrike air traffic control specialist train-
ees: III. Changes in demographic characteristics of
Academy entrants and biodemographic predictors of
success in air traffic controller selection and Academy
screening. (DOT/FAA/AM-90/4). Washington,
DC: Federal Aviation Administration Office of
Aviation Medicine. (NTIS No. ADA223480)

Dailey, J.T., 8 Pickrel, E.W. (1984). Development of
the air traffic controller Occupational Knowledge
Test. In S.B. Sells, J. T. Dailey, & E.W. Pickrel
(Eds.), Selection of air traffic controllers (pp. 299-
322). (DOT/FAA/AM-84/2). Washington, DC:
Federal Aviation Administration Office of Avia-
tion Medicine. (NTIS No. ADA147765)

Davis, J.A. (1971). Elementary survey analysis.
Englewood Cliffs, NJ: Prentice-Hall.

Della Rocco, P.S. (1998). Air traffic control specialise
screening programs and strike recovery. In D.
Broach (Ed.), Recovery ofthe FAA air traffic control
specialist workforce, 1981-1992 (pp. 17-22). Wash-
ington, DC: Federal Aviation Administration
Office of Aviation Medicine.



DOT/EAA/AM-99/16

Office of Aviation Medicine
Washington, D.C. 20591

DTIC QUALITY INSPECTED §

Differential Prediction of FAA
Academy Performance on the
Basis of Race and Written Air
Traffic Control Specialist
Aptitude Test Scores

Dana Broach

William L. Farmer

Willie C. Young

Federal Aviation Administration
Civil Aeromedical Institute
Oklahoma City, Oklahoma 73125

May 1999

Final Report

This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.

@

U.S. Department
of Transportation

Federal Aviation
Administration



FULL vs. BASIC MODEL TEST

[REGRESS
(DEPENDENT=SCREEN
METHOD=ENTER TMC
IMETHOD=ENTER TMC RACE TMC_RACE

Change in At
signifieant

PULL vs. SUBGROUP MODEL TEST:
Different stoopes?

REGRESS

IDEPENDENT=SCREEN
IMETHOD=ENTER TMC RACE TMC_RACE|
IREMOVE=RACE_TMC

‘Change in Monsighticant
significant change in A

vs. CROSSPRODUCT MODEL TEST BASIC vs. SUBGROUP MODEL TEST:
Different intercepts? Different intercepts?

REGRESS REGRESS
(OEPENDENT«SCREEN IDEPENDENT=SCREEN
IMETHOOSENTER

Figure-4: Step-down hierarchical regression analysis logic and SPSS syntax



Our step-down analysis was conducted as follows,
racing che SPSS-KTM GPSS Inc., 1989} regression pro-
cedure. First, SCREEN was regressed on TMC only
(basic model). Second, the criterion was regressed on
TMC, the dummy coded group membership variable
(0 for whites, 1 for African Americans; in accordance
with Pedhazur (1982, p. 274) and the cross-product
of TMC and that dummy coded variable (full model).
This full model was cested against the simple model
of criterion and predictor test only for an incremental
change in the R® (goodness-of-fit index). A signifi-
cant change in R° suggested potential bias and dic-
tated that further testing for slope and/or intercept
differences for the groups be done.

Third, to test for slope differences between groups,
SCREEN was regressed on TMC and the group
membership variable (group model), and compared
with the full model. A significant increment in the R°
based on a comparison of the group to full model
implied different slopes.

Fourth, if slope differences were found, then
SCREEN was regressed on TMC and the cross-
product of aptitude and group membership (cross-
product model). The cross-product model was then
compared with the groug model; a significant change
in R° indicated intercept, as well as slope differences,

between groups. If no slope differences were found,
then the cross-product model was compared with the
basic model; a significant change in R’ indicated only
intercept differences between groups. The general
logic and associated SPSS-XTM syntax for the step-
down hierarchical regression analysis are illustrated

im Figure 4.

Technical feasibility

Statistical considerations. Restriction in range,
statistical power, and criterion bias are consider-
ations that must be explicitly considered in determin-
ing the technical feasibility of an investigation of
differential prediction under the Uniform Guidelines
(2 CER 1607.14.B. (8).{0 and (2; 29 CFR
1607.16.U)- Both explicit and incidental restrictions
in range are recurrent problems in ATCS selection
research, as evidenced by the sample sizes and de-
scriptive statistics in Table 3. Variance in TMC for
the research sample was explicitly restricted in range
due to selection on TMC. Therefore, correlations
between TMC and the SCREEN Criterion were cor-
rected with respect to the reference sample of 170,578

applicants, using the formula presented by Ghiseili,
Campbell, and Zedeck (1981, p. 299). Correlations
between variables indicating minority status and the
criterion were incidentally restricted in range. These
minority status-criterion correlations, including the
minority status-by-predictor cross-product to
SCREEN correlation, were corrected with respect to
the reference sample of 170,578 applicants using the
Ghiselli, Campbell, and Zedeck (p. 304) formula for
incidental range restriction. Separate differential pre-
diction analyses were conducted based on sample and
corrected correlations, as required by the Uniform
Guidelines (29 CFR 1607.15.B. (8)).

The samples in this analysis were of sufficient size
to provide enough statistical power to detect even
small effects associated with group membership. The
step-down approach pools majority and minority
data in a single sample to test the null hypothesis that
a common regression line provides the best fit to the
data. We estimated the available statistical power
using Cohen’s (1988) regression. power tables for
three independent variables at an alpha of .05. The
tisk of a type Il error (failing to find an effect, that in
fact, was present) was very low, with a .995 probabil-
ity of detecting even very small effect sizes (P < .01,
or R = .14) with a sample of more than 8,000 cases.

Criterion considerations. Finally, as noted by Sackett
and Wilk (1994), Laucenschlager and Mendoza (1986),
and the Uniform Guidelines (29 CFR 1607.16.U), the
technical feasibiliry of an assessment of differential
prediction depends upon the quality of the job-relevant
criterion. Ifthecriterion was systematically biased against
minority members, fot example, then the regtession-
based method could not be used to determine the
presence or absence of differential prediction by sub-
group. Descriptive statistics for the criterion are pre-
sented in Table 4; the distribution of criterion scores by
minority status is shown in Figure 5. Observed mean
criterion score differences in SCREEN by race were
about .76 SD for the research sample, about rwice the
estimated differences of 3 to 4 SD by race reported by
Ford, Kraiger, and Schectman (1986) for ratings-based
criteria. In view of the large mean score difference on the
composite SCREEN criterion score, we conducted a
secondary analysis of the components of that score by
race. The principal components of the SCREEN com-
posite score were (a) the average percentage scor¢on four
25-item multiple choice and one map knowledge tests,
known as the academic block averagt score


Table 2

ATCS Aptitude test battery scoring

Test OPM # Scoring Weight N Items
MCAT 510 N Right 2 110
ABSR 157 N Right - (0.25*N Wrong) 1 50
TWws ((2*MCAT) + ABSR]*

Notes: *The TWS raw score was then transformed by an OPM transmutation tabie to the TMC score with a mean of

70 and maximum of 100.

TMC was used in our differential prediction analy-
ses as the measure of candidate aptitude, as it pro-
vided a measure of ability unadjusted for previous
experience, occupational knowledge, and/or military
service. Descriptive statistics for the predictor scores
are presented in Table 3 for the applicant sample, all
FAA Academy entrants, and the research sample.
Minority status was represented by the independent
variable RACE, coded as 0 = Whites, 1 = African
Americans. The interaction term (TMC*RACE) was
computed as the cross-product of RACE and TMC.
Descriptive statistics for the predictor by race are
presented in Table 4. The mean TMC for African
Americans (89.77, SD = 5.53) was significantly less
than the mean TMC for Whites (M = 91.67, SD =
4.77; 8,822) = 6.56, p < 001). The distribution of
TMC scores for Whites and African Americans is
illustrated in Figure 3.

C-erion. The criterion in the differential predic-
tion analysis was performance in the FAA Academy
initial controller training program, known as the
ATCS Non-radar Screen (“the Screen”). The Screen
was originally established in response to recommen-
dations made by the U.S. Congressional House Com-
mittee on Government Operations (U.S. Congress,
1976) to “...provide early and continued screening to
insure the prompt elimination of unsuccessful train-
ees and relieve the regional facilities of much of this
burden” (p. 13). The Screen was based upon a min-
iaturized training-testing-evaluation personnel selec-
tion model (Siegel, 1978, 1983; Siegel & Bergman,
1975) in which individuals with no prior knowledge
of an occupation are trained and then assessed for
their potential ro succeed in the job. Thirteen assess-

ments of performance, including six classroom tests,
observations of performance in six laboratory simu-
lations of non-radar air traffic control, and a final
written examination, were made during the Screen
(Della Rocco, 1998; Della Rocco, Manning, & Wing,
1990). The final summed composite score (SCREEN)
was weighted 20% for the classroom tests, 60% for
laboratory simulations, and 20% for the final exami-
nation. A minimum SCREEN score of 70 was required
to pass. The final composite score was the criterion
measure in this study. Descriptive statistics for SCREEN
scores are also presented in Table 3 forall FAA Academy
Screen entrants and for the research sample.

Procedure

Regression analysis. The classical, regression-based
model of test bias was used as our analytic framework
to evaluate the degree to which the written ATCS test
battery differentially predicted performance in the
Screen. Step-down hierarchical multiple regression
analysis (Lautenschlager & Mendoza, 1986) was used
to evaluate test bias, The step-down approach overcomes
the shortcomings of the various step-up procedures
(Bardlett, Bobko, Mosier, & Hannan, 1978; Cohen
& Cohen, 1975) by accounting for the various changes
in the sum of squared error term incrementally, while
at the same time ensuring more statistical power than
the other methods (Lautenschlager 8& Mendoza).
Step-down analysis assumes the null hypothesis that
a common regression line provides the best least-
squares fit to the data. The alternative is that a full
model including slope and intercept differences be-
tween groups is required to fit the data.


Table 9
Crosstabulation of dichotomized predictor scores (TMC <90, 290) and FAA Academy
Screen outcomes (SCREEN <70, >70) by race

African Americans
Fail Pass
__ (SCREEN < 70) (SCREEN > 70) _ Row Totals
Hire
(TMC > 90) n 84 70 154
Row % 54.5% 45.5%
Column % 48.6% 64.2% 54.6%
Total % 29.8% 8% Cire)
False Positive True Positive
No Hire
(TMC <90) n 89 39 128
Row % 69.5% 30.5%
Column % 51.4% 35.8% 454%
Total % 31.6% : 13.8% (No Hire)
True Negative False Negative
Column Totals 173 109 282
61.3% 38.7%
(Fail Screen) (Pass Screen)
Whites
Hire
(TMC > 90) n 1,768 3,728 5,496
Row % 32.2% 67.8%
Column % 56.5% 68.9% 64.3%
Total % 20.7% 43.6% (Hire)
False Positive True Positive
No Hire
(TMC <90) n 1,362 1,684 3,046
- Row % 44.1% 55.3%
Column % 43.5% 31.1% 35.7%
Total % 15.9% 19.7% (No Hire)
True Negative False Negative
Column Totals 3,130 5,412 8,542
36.6% 63.4%

(Fail Screen) (Pass Screen)

19



VanDeventer, A.D. (1983). Biographical profiles of
successful and unsuccessful air traffic control spe-
cialist trainees. In A.D. VanDeventer, D.K. Tay-
lor, W.E. Collins, & J.O. Boone (Eds.), Three
studies of biographical factors associated with success
in air traffic control specialist screening/training at
the FAA Academy (pp. 1-5). (DOT/EAA/AM-83/
6). Washington, DC: Federal Aviation Adminis-
tration Office of Aviation Medicine. (NTIS No.
ADA128784/6)

Wigdor, A.K., & Garner, W.R. (Eds.) (1982). Ability
testing: Uses, consequences, and controversies: Part
1. Report of the committee. Washington, DC: Na-
tional Academy Press.

25

Wigdor, A-K., & Sackett, P.R. (1993). Employment
testing and public policy: The case of General
Aptitude Test Battery. In H. Schuler, J.L. Farr, &
M. Smith (Eds.), Personnel selection and assess-
ment: Individual and organizational perspectives
(pp. 183-204). Hillsdale, NJ: Erlbaum.

Zeisel, H. (1957). Say it with figures. (4th Ed.). New
York: Harper & Row.



DIFFERENTIAL PREDICTION OF FAA ACADEMY PERFORMANCE ON THE Basis OF
RAcE AND WRITTEN AIR TRAFFIC CONTROL SPECIALIST APTITUDE TesT SCORES

The Federal Aviation Administration (FAA) is
committed to attracting, retaining, developing, and
managing a productive and skilled work force that
visibly reflects the nation’s diversity (FAA, 1993,
1998). Achieving this goal will require substantial
changes in the demographic profile of the Air Traffic
Control Specialist (ATCS) workforce, the single larg-
est (17,000) and most publicly visible occupational
group in the agency. Air traffic control isa career field
in which minority workers have been historically
under-represented. From 1981 through February
1992, entry into the occupation was determined by
applicant performance on a’ written aptitude test
battery (Aul, 1991). This test baccery emphasized the
organization, definition, and manipulation of the
perceptual field through verbal and numeric reason-
ing (Harris, 1986). Our purpose in this paper was to
examine the fairness of the Office of Personnel Man-
agement (OPM) written ATCS aptitude test battery
as the first step toward assessing to what degree, if
any, the battery may have served as an “engine of
exclusion” (Seymour, 1988) of minorities from the
ATCS occupation.

By fairness, we are explicitly referring to the regres-
sion model of test bias (also referred to as the “Cleary
model,” “predictive bias,” and “differential predic-
tion”) for which there is a reasonable professional
consensus, as embodied in the current Standards for
Educational and Psychological Testing (“Standards”;
American Educational Research Association, Ameri-
can Psychological Association, & National Council
on Measurement in Education, 1985) and the Prin-
ciples for the Validation and Use of Personnel Selection
Procedures (“Principles”; Society for Industrial and
Organizational Psychology (SIOP), 1987, p. 18). As
noted by Sackert (1996), it is important to differen-
tiate between predictive bias as a technical character-
istic of the use ofa test score ina particular setting and
fairness as 2 value judgment about the pattern of
outcomes arising from use of the test score. Our focus
in this paper is specifically on predictive bias as a

technical characteristic of a composite score used for
competitive selection of applicants into the ATCS
occupation between 1981 and 1992.

Under the Uniform Guidelines on Employee Selec-
sion Procedures (* Uniform Guidelines; "29 CFR 1607,
Equal Employment Opportunity Commission,
1978), an investigation of predictive bias encom-
passes two issues. First, the impact on protected
groups arising from use of a particular cut score on
the predictor, must be evaluated. A selection rate for
any protected group that is less than four-fifths (4/5
or 80%) of that of the majority group will “...gener-
ally be regarded by the Federal enforcement agencies
as evidence of adverse impact” (29 CFR 1607.
14B.(8).(o)). Second, where use of a selection proce-
dure results in adverse impact, the Uniform Guide-
kines require that the user of the test evaluate the
degree to which differential predictions of future job
performance are made from selection test scores by
subgroup (29 CFR 1607.14.B.(8).(b)). A test exhib-
its predictive bias under the Uniform Guidelines if“...
members of one race, sex, or ethnic group character-
istically obtain lower scores on a selection procedure
than members of another group, and the differences
in scores are not reflected in differences in 2 measure
of job performance” (29 CFR 1607.14.B.(8).(a)). In
other words, a test demonstrates predictive bias “...
if the criterion score predicted from the common
regression line is consistently too high or too low for
members of the subgroup” (Cleary, 1968, p. 115).

A formal analysis of the controller apticude test
battery’s impact on minority applicants was not tech-
nically feasible, as racial identifiers were aot collected
from ATCS job candidates at the time of testing.
However, previous research reported significant dif-
ferences in pass rates between whites and African
Americans (Rock, Dailey, Ozur, Boone, & Pickrel,
1984a). In view of this previous research, an investi-
gation of the relationship between test scores and job
performance for evidence of differential prediction,
in accordance with the Uniform Guidelines (29 CFR


performance in the Screen to subsequent job outcomes
such as performance in radar training and attainment of
FPL status has been repeatedly demonstrated (Broach,
1998: Broach & Manning, 1994; Della Rocco, Man-
ning, & Wing, 1990; Manning, Della Rocco, & Bryant,
1989). These findings indicate that the Screen was
relevant to the ATCS job as required by the Uniform
Guidelines, and was, therefore, an appropriate criterion
for this investigation of differential prediction on the
basis of written ATCS aptitude test scores.

RESULTS

Without corrections for restriction in range
Computed correlations, without corrections for
restriction in range, are presented in the lower left-
hand corner of the matrix in Table 6. TMC was
significantly correlated with minority status (RACE;
r = -0.070, p < .01), the predictor-group cross-
product (TMC*RACE; r = - .057, p S$ .01), and the
final score in the FAA Academy Screen (r= .199, p<
.001). RACE was negatively correlated with the cri-
terion SCREEN score (r = -.133, p < .001). The
results of the differential prediction analysis, using
the step-down hierarchical regression analysis on the
basis of the sample correlation matrix without correc-
tions for restriction in range, are presented in Table 7.

Table 6

The null hypothesis thac a common regression line
provided the best fit was rejected in the basic versus full
model analysis, suggesting the presence of some degree
of test bias. The incrementin R’ gained by using the full
model (predictor, group membership, and cross-prod-
uct) rather than the basic model (predictor only) was
significant (AR? = .015, AF = 68.34, p 5.001). Next, the
null hypothesis of same slopes by race could not be
rejected in the full versus subgroup model analysis. The
subgroup model (predictor and group membership) did
not explain any less variance than the full model (AR? =
0, AF= 2.96, ns). Following the analytic logic illustrated
in Figure 4, the basic and subgroup models were next
compared to determine if the intercepts were different
for African-Americans and whites. The null hypothesis
of same intercepts was rejected, with removal of RACE
leading to a significant reduction in explained variance
(AF? = -.014, AF = 133.69, p<.001). The regression of
SCREEN on TMC for African Americans and whites is
plotted in Figure 6. Overall, the results obtained with
the uncorrected correlations did not indicate the
need for separate regression equations for African
Americans and whites as the slopes were the same for
the two groups. However, as shown in Figure 6,
criterion scores predicted from the white regression
line would consistently over-predict criterion scores
for African Americans.

Sample (lower comer) and corrected (upper corner) correlation
matrix for differential prediction analysis by race

TC RACE TMC*RACE SCREEN
TMC 0.070 0.057 0.519
RACE -0.070** 0.998 -0.208
TMC*RACE — -0.057** 0.9980"** 0.191
SCREEN 0.199** = 9.133" = -9,132""*
“p<.01, “*p<.001
12


Group

¢ African American

White

TMC

100

Figure 6: Regression of SCREEN on TMC for African Americans and whites, without corrections for

restriction in range

Moreover, the regression line for whites is essen-
tially the same regression line for the combined
sample. The regression (without corrections for re-
striction in range) of SCREEN on TMC for the
combined sample is

PREDICTED SCREEN = 29.04 + (0.467*TMC). (1)

The regression equation (without corrections for
restriction in range) for whites only is
PREDICTED SCREEN = 30.28 + (0.456*TMC). (2)

The constant for the whites-only equation is within
one standard error of the value computed on the basis
of che combined sample; similarly, the unstandardized
regression weight computed for whites only is within

14

one standard error of the value computed for the
combined sample. While no formal statistical test is
possible as the samples (combined and whites only)
are not independent, we argue that the combined
group regression line is, in essence, identical to the
regression line for whites. Determining the weights
for the aptitude test components and cut score onthe
basis of a combined sample that is predominantly
white, as was done for the ATCS aptitude test battery
(Rock, Dailey, Ozur, Boone, & Pickrel, 1982, p.
193), in effect sets the weights and cut score on the
basis of the white regression line. As a consequence,
it is appropriate to assert that the performance of
African Americans is over-predicted on the basis of
the white regression line, and the test i: biased under
Cleary’s definition.



Technical Documentation Page"

1, Report No. 2. Government Accession No. 3. Recipient's Catalog No.
DOT/FAA/AM-99/16

4. Title and Subtitie 5. Report Date

Differential Prediction of FAA Academy Performance on the Basis of May 1999

Race and Written Air Traffic Control Specialist Aptitude Test Scores 6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.
Broach, D., Farmer, W. L., & Young, W.C.

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

FAA Civil Aeromedical Institute

P.O. Box 25082

Oklahoma City, OK 73125-0082 11, Contract or Gront No.

12. Sponsoring Agency Name ond Address 13. Type of Report and Period Covered
Office of Aviation Medicine

Federal Aviation Administration

800 Independence Avenue, S. W.

‘Washington, DC 20591

15. Supplementary Notes
This research was conducted under task AM-98-B-HRR-509.

16. Abstract -

The written air traffic control specialist (ATCS) aptitude test battery was evaluated for evidence of predictive bias
within the framework of the Uniform Guidelines on Employee Selection Procedures (29 CFR 1607) in a retrospective
analysis. Step-down hierarchical regression analysis (Lautenschlager & Mendoza, 1986) was used to investigate
differential prediction of performance in initial ATCS training at the Federal Aviation Administration (FAA)
Academy in a sample of 282 African-American and 8,542 white first-time competitive entrants. Analysis based on
correlations without corrections for restriction in range found significant differences in the intercepts, but not
slopes, for African Americans and whites. Analysis based on correlations, corrected for explicit and implicit
restriction in range, found significant differences in slopes and intercepts by race, suggesting that separate regression
equations were appropriate to predict Academy performance for the groups. The two analyses indicated that the
composite score on the written ATCS test battery exhibited predictive bias as defined by the Uniform Guidelines on
Employee Selection Procedures (29 CFR 1607) and Cleary (1968). Specifically, the composite score TMC over-
predicted the performance of African Americans in initial training at the FAA Academy. As a consequence of the
over-prediction, significantly more of the African Americans that were accepted into training for the ATCS
occupation on the basis of their aptitude test scores went on to fail training than would have been expected on the
basis of the common or majority (white) regression line. An alternative explanation is considered that the observed
differential prediction reflected criterion bias or other group differences in factors such as educational achievement
and age. A path analytic approach is outlined for investigating the complex interactions between test score, the
criterion, race, education, and age. Additional research on the consequences of over-prediction for African
Americans in the FAA Academy is recommended in closi

17. Key Words. 18. Distribution Statement
ATCS selection, Test Bias, Adverse Impact, Document is available to the public through
Fairness, Equal Employment Opportunity, the National Technical Information Service,
Personnel Selection, Predictive Bias Springfield, Virginia 22161
19. Security Classif. (of this report) 20, Security Classif. (of this page) 21. No. of Pages 22. Price
UNCLASSIFIED UNCLASSIFIED 28
Form DOT F 1700.7 (8-72)

Reproduction of completed page authorized



Della Rocco, P.S., Manning, C.A., & Wing, H. (1990).
Selection of controllers for automated systems: Appli-
cations from current research. (DOT/FAA/AM-
90/13). Washington, DC: Federal Aviation Ad-
ministration Office of Aviation Medicine. (NTIS
No. ADA230058)

Dunbar, S.B., & Novick, M.R. (1988). On predicting
success in training for men and women: Examples
from Marine Corps clerical specialties. Journal of
‘Applied Psychology, 73, 545-550.

Equal Employment Opportunity Commission (1978).
Uniform Guidelines on Employee Selection Proce-
dures, 29 CFR 1607.

Federal Aviation Administration. (1993). FAA diversity
plan. Washington, DC: Federal Aviation Admin-
istration Office of the Administrator for Human

"Resources Management.

Federal Aviation Administration. (1998). 1998 Federal
Aviation Administration strategic plan. Washing-
ton, DC; Federal Aviation Administration Office
of the Administrator.

Ford, J.K., Kraiger, K., & Schectman, S.L. (1986).
Study. of race effects in objective indices. and
subjective evaluations of performance: A meta-
analysis of performance criteria. Psychological Bul-
letin, 99, 330-337.

Ghiselli, E., Campbell, J., & Zedeck, S. (1981). Mea-
surement theory for the behavioral sciences. San
Francisco: Freeman.

Gordon, R.A., Lewis, M-A., & Quigley, A.M. (1988).
Can we count on muddling through the gcrisis in
employment? Journal of Vocational Behavior, 33,
424-452,

Harris, P. (1986).A construct validity study of the Federal
Aviation Administration Multiplex Controller Ap-
titude Test. Washington, DC: U.S. Office of Per-
sonnel Management Office of Staffing Policy.

Hartigan, J.A., & Wigdor, A.K. (1989). Fairness in
employment testing: Validity generalization, minor-
ity issues, and the General Aptitude Test Battery.
Washington, DC: National Academy Press.

23

Hunter, J.E. (1983). Fairness of the General Aptitude
Test Battery: Ability differences and their impact on
minority hiring rates. (United States Employment
Service Test Research Report No. 46). Washing-
ton, DC: U.S. Department of Labor Employ-
ment and Training Administration.

Hunter, J.E., Schmidt, F.L., & Rauschenberger, J.
(1984). Methodological and. statistical issues in
the study of bias in mental testing. In C.R:

Reynolds & R.T. Brown (Eds.), Perspectives on
bias in mental testing. New York: Plenum.

Kraiger, K., & Ford, J.K. (1985). A meta-analysis of
ratee race effects in performance ratings. Journal

of Applied Psychology, 70, 56-65.

Lautenschlager, G.J., & Mendoza, J.L. (1986). A
step-down hierarchical multiple regression
analysis for examining hypotheses about test
bias in prediction. Applied Psychological Mea-
surement, 10, 133-139.

Lee; R., Miller, K., &¢ Graham, W. (1982). Corrections
for restriction of range and attenuation in crite-
tion-related validation studies. Journal of Applied
Psychology, 67, 637-639.

Lewis, M.A. (1978). Objective assessment of prior air
traffic control related experience through the use
of the Occupational Knowledge Test. Aviation,
Space and Environmental Medicine, 49, 1155-
1159.

Lilienthal, M.G., 8& Pettyjohn, F.S. (1981). Multiplex
Controller Aptitude Test and Occupational Knowl-
edge Test: Selection tools for air traffic controllers.
(NAMRL Special Report 82-1). Pensacola, FL:
Naval Aerospace Medical Research Laboratory.
(NTIS No. ADA118803).

Linn, R_L. (1994), Fair test use: Research and policy. In
M.G. Rumsey, C.B. Walker, 8 J.H. Harris (Eds.),
Personnel selection and classification (pp. 363 -
375). Hillsdale, NJ: Erlbaum.

Manning, C.A. (1991). Procedures for selection of air
traffic conttol specialists. In H. Wing & C.A.
Manning (Eds.), Selection of air traffic controllers:
Complexity, requirements, and public interest (pp.
13-22). (DOT/FAA/AM-91/9). ‘Washington,
DC: Federal Aviation Administration Office of
Aviation Medicine. (NTIS No. ADA238267)



NOTICE

This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents or use thereof.


~ White
-e- African-American|"

1600 ----

1400

8

Whites (N)
co]
8

600 +

2
3

African-Americans (N)

2 30 35 40 45 50 §5 60

65 70.75 80 8&5 90

95

SCREEN
Figure 5: Criterion (SCREEN) score distribution by race

(BLOCKAVG), (b) the percentage score on a 50-item
comprehensive multiple choice knowledge test known
asthe Comprehensive Phase Test (CPT), (c) the average
score for the best five of six graded laboratory exercises
{AVGLAB5), and (d) the percentage score on a 100-
item comprehensive multiple choice Controller Skills
Test (CST). Additional details ling these mea-
sures can be found in Della Rocco (1998), Della Rocco,
Manning, and Wing (1990), and Manning, Della Rocco,
and Bryant (1989). The results of the analysis of com-
ponent scores by race are presented in Table 5. African
American students scored significantly less well than
white students on the four criterion component scores.
So it might be argued that the statistically significant
mean differences on the overall criterion and its
components indicated “systematic bias” in the crite-
rion against minority members, and the differential
prediction analysis was technically infeasible under
the Uniform Guidelines.

10

The relevance of criteria are of particular concern
when there are significant differences berween groups
on those criterion measures (29 CFR 1607.14.B.(2)).
According to the Uniform Guidelines at 29 CFR
1607.14.B.(3), “Where performance in training is used
as a criterion, success in training should be properly
measured and the relevance of the training should be
shown either through acomparison of the content of the
training program with critical or important work
behavior(s) of the job(s), or through a demonstration of
the relationship between measures of performance in
training and measures of job performance.” The objec-
tive tests and laboratory simulations in the Screen were
explicitly linked to specific air traffic control facts,
definitions, and procedures found in the air traffic
procedures manual. Many of the tasks taught in the
Screen were comparable to the tasks performed by sector
controllers in en route centers (Della Rocco, Manning,
& Wing, 1990). In addition, the relationship of


*siskjeue worssosdor

100° > 4,,, Teoqyoseso1y umop-dois Jo Moy pue 2180] 405 | o1n8,y 29g “payonpuod you WosURduIOD JopoUs ONpord-sso1D “A [NY :s910N
wand] DOE 04469 CE1 $10'0- 0r0'0 OWL
aasV9 1ST 9S0'0 aOva + OWL sydaoruy :dnosBqng ‘A o1seg,
youpoid-ssox) “A {Md
wna 1ST 96% 000'0 S00 dova + OWL
40x L891 S00 SOVU«OWL + SOVE + OWL sedojg :dnox8qng ‘a jing
4ne6L'891 ans VE'89 S100 soo AOVUOWL + FOVA + OWL
san CPOE 000 OWL SBIq [IBI9AO :][M.J “A SISUg
d dv av a 1opow sisXjeuy

eBues ul uonoyyses

40} SUO}9E109 INOUYIM XUJEL UOHE{ELI09 Jo s!seq UO eoB: Aq seIq 189} 40) siskjeue UO}ssesBel jeoIyosBIE14y UMOP-do)S Jo SyNsEL

Leen

13