courses for the En Route and Terminal options
were essentially based upon a miniaturized
training testing personnel selection model (Siegel,
1978, 1983; Siegel & Bergman, 1975) in which
individuals with no prior knowledge of the
occupation could be assessed for their potential to
succeed in air traffic control. Individuals who
failed were separated from the occupation.
Separate En Route and Terminal Academy
programs were administered until October 1985.
All candidates included in the present study were
graduates of these dual programs. Each of the
screening courses, the En Route Initial Qualifica-
tion Training and the Terminal Initial Qualifica-
tion Training, trained and assessed students on
the application of nonradar procedures in each
type of airspace, respectively. Although there
were a number of specific differences between
the 2 programs, both generally consisted of an
academic and 2 laboratory phase. Didactic class-
room training on nonradar air traffic control
{ATC) rules and principles was given in the
academic phases of the optionspecific courses.
Academic content included aircraft separation
Tules, cooperative agreements, phraseology, and
flight progress strip-marking. Students then
applied these rules, principles, and procedures of
ATC in 2 series of thirty-minute scripted scen-
arios in the laboratory phases of the dual courses
of instruction. Students were evaluated in acade-
mics based upon muitipie choice examinations
and a map test of the synthetic airspace. In the
laboratory, students were graded for their perfor-
mance on 6 scripted scenarios. A final timed,
paper-and-pencil exam assessed application of
ATC rules. A composite of these scores deter-
mined pass/fail status. The composite score was
heavily weighted on laboratory performance. A
successful candidate was required to achieve a
final composite score of 70 or better.

Performance in the ATCS screening courses
appears to predict future performance in field
training reasonably well. For exampie, Manning,
Della Rocco, & Bryant (1989) reported a raw
correlation of -.24 (p < .01, N = 2,992) be-
tween the En Route Initial Qualification Course
final composite scores for 1981-1985 Academy
graduates and status in En Route field training.
Field training status in that study was coded 1 =
Full performance level, 2 = In training in or-

16

iginal option, 3 = Switched options, and 4 =
Failed. Correcting for range restriction due to
explicit selection on the predictor resulted in a
corrected correlation of -.44 between En Route
Initial Qualification Course scores and field
training status. Manning et al., also examined the
relationship of the Terminal Initial Qualification
Course composite final scores to Terminal train-
ing performance. They divided Terminal con-
trollers into 2 groups for that analysis: those
assigned te Terminal facilities without radar
(visual flight rules, or "VFR"; N = 441); and
those assigned to Terminal facilities with radar
("Radar"; N = 966). While the raw correlations
between Terminal course scores and status in
VFR training was non-significant (r = -.08, p >
.01), the initial qualification course score was
significantly related to various indices of time in
training at specific positions within the tower
cab. In contrast, status in the Radar Terminal
training program was predicted reasonably well
by the Terminal Initial Qualification Course final
composite scores (7 = -.28, p < .01, N = 962).
Adjusting for restriction in range due to explicit
selection on the predictor improved that overall
correlation to -.5i between Terminal course
scores and Radar Terminal training status. Over-
all, the FAA Academy ATCS option-specific
Screening programs appeared to predict field
training status reasonably well. In other words,
developmental controllers who performed well!
in the FAA Academy programs appeared to go
on to perform well in technical field training.
While supporting data are not easily available, it
may be reasonably supposed that persons who
did well in the ATCS initial qualification courses
and field training were also likely to do well
upon reaching full performance level.

This sophisticated and rigorous screening and
on the job training system assured the tech-nical
competence of the ATCS journeyman or full
performance level (FPL) workforce. Systematic
selection of first line supervisors of controllers is
accomplished through the SIDP. These supez-
visors, as previously noted, oversee controller
teams responsible for air traffic operating witnin
contiguous areas or sectors of airspace. For
example, first-line En Route ATCS supervisors,
or Area Supervisors, carefully monitor current
and anticipated activities in the assigned sectors


Table 6

Results of regression analyses using

field training performance measures as predictors of

component rating dimension scores in SIDP En Route controllers

Criterion Predictors R R 8 F Pp
PSA Ratings
Composite OST Hours 23 05 ~.23 16.13 -0001
Communication OJT Hours 18 03 -.18 9.35 -003
Interpersonal OST Hours 18 .03 -.18 9.57 003
Leadership OFT Hours 24 06 ~.24 16.91 0001
Technical OJT Hours 32 10 -.25 16.60 0001
Inst Rating +15
SBI Ratings
Ag Programs ns
Stpervisory Role Years to FPL 14 02 -14 5.35 03
Problem Solving ns
Judgment ns
Decisiveness OJT Hours 12 01 -.12 4.14 05
Organizing ns
Tnterper Skill ns
Communication ns
Direction/Motiv ns



REFERENCES

Della Rocco, P. S., & Broach, D. (1992). Air
traffic control specialist and technical
competence in initial training and selection as
a first-line supervisor. In J. G. Myers (Ed.),
A longitudinal examination of applicants to
the Air Traffic Supervisory Identification and
Development Program. Washington, D.C.:
FAA Office of Aviation Medicine Report No.
DOT/FAA/AM-92/16.

Manning, C. A. (1991). ATCS field training
performance and success in a supervisory
selection program. In J. Myers (Chair), Paths
to success: A longitudinal examination of air
traffic controi first-level supervisors. Sym-
posium presented 2: the Sixth International
Symposium on Aviation Psychology, Colum-
bus, Ohio.

Manning, C. A., Della Rocco, P. S., & Bryant,
K. D. (1989). Prediction of success in FAA
air traffic control field training as a function
of selection and screening test performance.
Washington, D.C.: FAA Office of Aviation
Medicine Report No. DOT/FAA/AM-89/6.

Stusher, A., Van Dyke, J., & Rose, G., (1972).
Technical competence of group leaders,
managerial role, and productivity in engineer-
ing design groups. Academy of Management
Journal, 15, 197-204.

TABLE i
Relationship between overall instructor rating and SIDP status for all ATC options

SIDP Status M SD N
Referred 4.09 37 926
Not referred 3.96 85 293
Eligible 4.09 86 674
Not eligible 4.11 -88 248
Selected 4.23 81 166
Not selected 4.04 88 508
TABLE 2
Training success at first facility and SIDP status for al! ATC options
% % Not

SIDP Status Successful Successful N
Referred 82.1 17.9 1018
Not referred 82.2 17.8 331
Eligible 83.1 16.9 739
Not eligible 79.3 20.7 275
Selected 88.8 11.2 179
Not selected 81.3 18.8 560

33


G9 8=—SOE Sth 8°96 fly = 8°86 Cop s90l th 6S IOI €0S Goll sinoH LO
VOL LOPE 79II LOL §=¢80l SUS LOW =769F «6 =U Lis. T'¥9l skeq {0
e se e 9 € x y L v L t u “Fda 0} 8180),
8 vr OT ee 6 vy 6 ev 6 ev 8 ov Supey sojnsuy
as Ww as nN as Ww as Ww as Ww as Ww sainseay Suyuyesl, pred
powe]eg pa19[9g 10N 214131 9141811g 30N pougjoy pe19jou ON
uonefeg Asostasadng uoperepisuog JeyN, IAS 0} porajoy
Joy 9118119

S4a]JOMUOI GUD 44aMOL AQIS UI snivis puv aouvuuofied Sujuyo.s pjayf uaansaq dyysuopwjay

TT 198L

42



TABLE 2
Correlations between the OPM selection tests, experimental tests
and selected SIDP ratings for applicants entering the en route option

Peer-Supervisory Assessment Ratings

OPM & EXPERI-
MENTAL TESTS COMP COMM INTR LEAD TECH SBI TOTAL

MCAT 05 03 03 04 .O7* -.05
TMC 04 03 02 03 .06* -.01
ABSR -.01 O01 -.01 -.01 -.01 10**
OKT -.00 -.02 -.03 .00 04 00
DHT 02 04 -.00 02 04 2
DRT .06 04 02 02 -16* -.08
ps 05 p< 01
TABLE 3

Correlations between the OPM selection tests, experimental tests and
selected SIDP ratings for applicants entering the terminal option

Peer-Supervisory Assessment Ratings

OPM & EXPERI-
MENTAL TESTS COMP __ COMM. INTR LEAD TECH _ SBI TOTAL

MCAT 03 03 -00 03 05 04
TMC 05 05 03 04 .05 .05
ABSR 05 .06* .08* .05 01 04
OKT ~.03 ~.02 ~.06* -.01 -.01 -.02
DHT +16 17 -.15 -.13 -.14 AS
DRT .21* .22** .18* 16 .22** ll

10


Table 2
Correlations between performance variables

1) PSA Composite
2) SBI Knowledge 13"
3) SBI Skill 07 arg
4) Performance Rating 25 0 03
5) Months to FPL -.04 -.03 +01 00
Measures 1 2 3 4 5

33

p< 0 W= 743)


defined.

Interviews with air traffic facility selecting
officials conducted during an evaluation of the
SIDP noted that factors other than SIDP perfor-
mance, such as diversity in work experiences and
references from applicants’ supervisors, play a
large part in the selection decision (Office of the
Associate Administrator for Human Resource
Management, 1991). Wentification of important
factors in the selection decision is needed as part
of the validation of the SIDP. In addition, collec-
tion of information on applicant experiences,
education, and other biodata might suggest what
additional types of factors may relate to success-
ful performance in SIDP. This type of informa-
tion could be provided to applicants to improve
their chances for success in the selection pro-
gram, as well as improving their effectiveness as
a supervisor.

Obviocsly, this study did not span the do-
main of possible predictors of success in SIDP
‘nd selection as a first-line supervisor. Future
-esearch in areas such as biodata, achievement
Motivation, and personality factors is needed to
contribute to the theoretical base for managerial
potential and assist in improving the operation
of SIDP.

REFERENCES

Levine, H. Z. (1986). Supervisory selection
systems, Personnel, 63, 61-67.

Myers, J. G., & Stutzman, T. M. Job task-
competency linkages among FAA first-level
supervisors. Washington, D. C.: Office of
Aviation Medicine Technical Report, DOT/-
FAA/AM-91/5.

Office of the Associate Administrator for Human
Resource Management. (1991). Supervisory
Identification and Development Program:
Selecting first-line supervisors in the FAA.
(Evaluation Report Number 91-1). Washing-
ton, D. C.: Federal Aviation Administration.

wn

Northrup, H. R., Cowin, R. M., Vanden Plas,
L. G., Fulmer, W. E., Bolick, R. E. Jr.,
Beliace, J. R., & Rosenzweig, A. H. (1978).
The objective selection of supervisors: A
study of informal industry practices and two
models for improved supervisor selection.
(Manpower and Human Resources Studies
No. 8). Philadelphia: University of Pennsyl-
vania, Wharton School of Business, Indust-
tial Research Unit.

NoruSis, M. J. (1990). SPSS advanced statistics
user’s guide. Chicago, IL: SPSS, Inc.

Phillips, 3. 5. (1985). Jmproving the supervisor’s
effectiveness. San Francisco: Jossey-Bass.

Rendero, T. (1980). Supervisory selection pro-
cedures. Personnel, 57, 4-10.

Siusher, A., Wan Dyke, A., & Rose, G. (1972).
Technical competence of group leaders. Ac-
ademy of Management Journal, 15, 197-204,

SPSS, Inc. (1990). SPSS reference guide. Chic-
ago, IL: SPSS, Inc.

U.S. Merit Systems Protection Board. (1989).
First-line supervisory selection systems in the
federal government. Washington, D. C.: U.
S. Government Printing Office. une)


were averaged for a PSA Composite score. Two
factors were found for the SBI ratings: a
knowledge factor made up of the 2 knowledge
rating dimensions (Knowledge of Supervisory
Role Performance and Organizational
Knowledge) and a skill factor made up of the
remaining seven rating dimensions. Because
ratings on the knowledge dimensions are based
on answers to questions rather than performance
in role-play scenarios, there was some concern
that the 2 factors reflected method variance
rather than separate dimensions. Thus, all items
were included in an item analysis using the
reliabilities procedure in SPSS® (Nye, 1990).
Item statistics suggested that the 2 SBI factors
should be kept separate for subsequent statistical
analyses. Ratings for each respective factor were
summed to create a SBI Knowledge Composite
and a Skill Composite. Correlations between the
composite variables and the technical
performance variables are shown in Table 2;
descriptive statistics for the variables for different
group comparisons are shown in Table 3.

Referral to the SBI

Analysis of variance (ANOVA) was used for
comparing mean differences between the Referred
and Not Referred groups. Only the Performance
Rating and Months FPL Experience were
included since the PSA composite operationally
defines who will be referred to the SBI and
would be expected to be lower for the Not
Referred group than the Referred group. The
Referred amd Not Referred groups had
significamiy different mean Performaive Katings
(F= 101.81, df=1,983, p<.001) but were not
significantly different on Months FPL Experience
(F< 1.0).

The probability of being referred to the SBI
was estimated using the SPSS® logistic regres-
sion procedure (Norusis, 1990). In the first
regression, the PSA Composite, Performance
Ratings, and Months FPL Experience were
entered into the regression equation using the
forward stepwise method. The second regression
excluded the PSA Composite. It was felt that
comparison of the 2 regressions would help to
identify the impact of the peripheral performance
variables on the probability of referral to the

49

SBI. Resuits of the first regression showed that
only the PSA Composite and Performance Rating
variables predicted the referral decision. The
goodness-of-fit was tested using -2 times the log
likelihood (-2 LL) of the model and indicated a
good fit to the data (-2 LL, x’(1,982)=555.42,
p=1). The overall classification rate was
89.85%. When the PSA Composite was excluded
from the equation, Performance Rating was stili
identified as a predictor. However, the fit to the
data was somewhat poor (-2 LL,
x°(1,983)=986.10, p=.47) and all cases were
predicted to be referred to the SBI.

Promotability

A multivariate analysis of variance test was
used to examine mean differences on the PSA
composite, Performance Rating, and Months FPL
Experience between those who were placed on
the EFC list (EFC) and those who were not (Not
EFC). SBI variables were excluded from this
analysis because the combination of rating scores
on the dimensions operationally determines who
is placed on the EFC list. The multivariate test
was not significant.

Again, 2 logistic regression analyses were
conducted. The first regression equation in-
cluded the SBI Knowledge and Skill Com-
posites, as well as the PSA Composite, Perfor-
mance Rating, and Months FPL Experience.
Only the SBI Knowledge and Skill Composites
predicted the promotability decision. The good-
ness-of-fit was tested using -2 times the log
iiketinOud (-Z LL) of the modei and indicated 2
good fit to the data (-2 LL, x"(1,742)=195.98,
p=1). The overall classification was 95.03%.
Given that the SBI ratings are used to opera-
tionally determine assignment to the EFC list,
this finding is not surprising. The second
tegression analysis excluded the SBI composite
variables; the PSA Composite and technical
performance variables did not enter into the
prediction of promotability status.

Selection
The dependent variables included in the

MANOVA were: SBI Knowledge Composite,
SBI Skill! Composite, PSA Composite, Perfor-


referral to the Skill-Based Interview.

Eligibility for Consideration. The probability
of making the “Eligibie for Consideration” list
after referral to the SBI was estimated by
entering the independent variables into the
equation, and evaluating mode! goodness-of-fit.
The analysis indicated that previous technical
training performancc did not predict making the
EFC list from the Skill-Based Interview. Assess-
ment of the goodness-of-fit using the -2 times the
log likelihood of the model indicated a poor fit to
the EFC data (En Route: -2 LL (348) =
399.2, p < .05; Terminal: -2 LL x(491) =
563.4, p < .05) under the null hypothesis that
tae modei did fit the data. The overall rate of
correct classifications by the model was 72.6%
for En Route and 73.4% for Terminal. However,
the model! predicted that all but 6 En Route and
1 Terminal applicant would have made the EFC
list. This resuit indicated that performance in
initial technical training failed to provide useful
information for the practical prediction of making
the EFC list from the interview process.

Selection as a supervisor. Finally, the prob-
ability of selection from the EFC list as a super-
visor was estimated by once again entering the
independent variables into the equation and
assessing model goodness-of-fit. This analysis
also indicated that previous performance in initial
training did not predict actual selection for a
first-line supervisory position in the air traffic
controller occupation. The null goodness-of-fit
hypothesis that the model fit the data for the
Terminal option was rejected based on -2 times
the log likelihood of the model (-2 LL x’G61) =
428.4, p <.01) despite an overall rate of 70.8%
correct classifications. This time the model
predicted that no cases would been selected as a
supervisor from the Terminal option EFC list.
For the En Route option, the null goodness-of-
fit hypothesis was not rejected based upon the -
2 times the log likelihood of the model (-2LL
(251) = 261.9, p < .31). Although the overall
correct classifications from the model was
78.6%, no selections from the EFC list were
predicted based upon this model using perfor-
mance in initial technical training. As with the
previous logistic regression analyses, the Acad-
emy measures of technical performance failed to

20

provide useful information for predicting selec-
tion as a supervisor.

DISCUSSION

The purpose of this study was to examine the
relationship of success in the Supervisory Iden-
tification and Development Program and prior
performance in the FAA Academy ATCS initial
qualification courses. Previous studies had dem-
onstrated significant correlations between Acad-
emy scores and subsequent field training perfor-
mance (Manning, Della Rocco, & Bryant, 1989;
Della Rocco, Manning, & Wing, 1990). The
basic premise that persons who perform well in
initial technical training for a technicatiy demand-
ing occupation would be more likely to be sel-
ected for first-line supervisory positions in that
occupation was tested from three perspectives.

The first research question examined the
relationship between Academy scores and the
ratings from the Peer/Supervisory Assessment.
Small, significant correlations were found be-
tween Academy scores and the PSA technical
competence assessment for the En Route option.
Thus, there was some evidence that En Rote
Academy scores were related to Full Pe: or-
mance Level (FPL) or iourneyman-levei perfor-
mance. The reason that the correlations were
small is not evident because greater correlation
has been reported between these Academy scores
and field training performance (Manning, et al.,
1989). However, candidates completed the
Academy up to 7 years prior to the assessment
for SIDP which is a considerable length of time
between the 2 sets of assessments. The predicted
relationship between Academy laboratory mea-
sures and PSA technical ratings was not found
for the Terminal option, however. This finding
corresponds to findings of Manning et al., (1989)
of the correlations between the Academy and
subsequent field training measures for terminal
option controllers.

The second research question assessed the
mean differences in Academy scores between
successful and unsuccessful applicants to SIDP
at 3 stages of the SIDP selection process. The
hypothesis that successful applicants would have
performed better in the Academy courses was not


“Le = v'09 Eby $99 ftp S99 LL? 60S £'0r 0°79 Vt £'% sMOH L{O
69L 666 L9 9°86 S69 = S'66 L's9 'L8 8°89 = 8°96 68s £68 shed Lf0
6 Lt 3 6} s cal oT Ll s sl or 07% “Tdel 0} S1b0, 8
s Ve e Or 6 or 8 o'r 6 Ov s I'v Supey soyonsU]
as Ww as WwW as Ww as Ww as WwW as Ww soanseoy SuyuyedL, Pint
Peypapes 2109]9§ JON 9118 21018114 10N POLIOJOY PesJ9j0y ON
uonsajeg Asostasodng uoHeJEp|suo? JoyN IGS 0} poxajoy
403 9191811

Suazjossuos yonosddy svpoy jouuuay dais uj snivis puo asouvuuofiad Sujuyo.r prayf ueamieg diysuopujay
L481


AN OVERVIEW OF THE AIR TRAFFIC CONTROL SPECIALIST AND
FIRST-LINE SUPERVISOR SELECTION SYSTEMS

Jennifer G. Myers, Ph.D.

The air tratfic control (ATC) job is unique in
several ways. It is one of the few federal jobs
where continued employment is based on the
successful completion of a selection and ‘raining
program wholely conducted by the government.
The skills necessary to perform the job are not
necessarily those acquired through formal voca-
tional or university training, although university-
based ATC training has recently been initiated on
a test basis. Like other technicai occupations,
reaching journeyman status is based on the
successive demonstration of the acquisition of
increasingly complex job skills.

Individuals interested in applying for an ATC
job first complete a battery of tests administered
by the Office of Personnel Management in
different regional locations. Eligibility for attend-
ing the FAA Academy Screen Program in
Oklahoma City is based on a composite of the
test battery scores, as well as meeting medical
and security screening criteria. Following the
successful completion of the Screen Program, in-
dividuals are assigned to a field facility for on-
the-job training. The length of field training
varies, depencing on the type of faci‘ty (en route
or terminal), and ranges from about 1 to 3 years
(see Figure 1).

Air traffic control specialists (ATCS) who
wish to advance into a supervisory position must
apply to the Supervisory Identification and
Development Program (SIDP). This is an agen-
cy-developed system for assessing individual
competencies and determining eligibility to
compete for first-line supervisory positions. The
SIDP was first proposed in response to the
recommendations of congressional committees
following the air traffic controller strike in 1981.
It was felt that by basing supervisory selections
on factors other than prior training and technicai
performance, the organizational culture that
precipitated the 1981 strike could be improved
and future labor-management problems could be
avoided. The SIDP was implemented on a iesi
basis in 2 regions, Southwest and Northwest

Mountain, in 1985 and continued until national
impiementation of the program in 1988. Since
1985, over 10,000 supervisors and non-super-
visors have applied to the program and approxi-
mately 5,000 of those applicants have successful-
ly completed the program.

Continued technical competence in air traffic
control is required in the supervisory position
because task performance includes supervision of
the technical werk of others. This entails a
requirement for maiataining operational cvrrency
vy performing in an operational position 16
hours per month. However, technical competence
is but one of the competencies assessed in the
SIDP. Rather, the SIDP is designed to allow
applicants to demonstrate other skills needed for
effective supervision of people and programs,
such as communications and interpersonal skills.
The SIDP is a multiple hurdle process for deter-
mining eligibility for supervisory positions (see
Figure 2).

Controllers must have completed one year at
the journeyman level and received a fully suc-
cessful rating on their last performance appraisal
to be eligible to apply to the prograza. The Peer-
Supervisory Assessment process requires the
SIDP applicant to identify from 4 to 7 peers
(depending on the size of the facility) and one or
more supervisors to rate him/her on 4 perfor-
mance dimensions: Interpersonal, Communica-
tions, Direction and Motivation, and Technical
Competence. A paired-comparison method is
used to rate each applicant against other ap-
plicants and benchmarks on each of the dimen-
sions. Supervisors and peer ratings are combined
to produce a single score for each dimension.
Dimension ratings are averaged to create a
composite score and the top applicants (based on
their composite score) are referred to the next
stage of the selection process, the Skill-Based
Interview (SBI).

The SBI is a combination of a face-to-face
interview and role play exercises. In the face-to-


the dependent variables. None of the measures of
field training performance sufficiently predicted
any stage of the SIDP process to justify includ-
ing them in a model.

Question 7: For tower cab controllers, in what
ways are the measures of training per-
formance related to the rating dimensions used
to determine success in SIDP?

Table 12 shows descriptive statistics for all
predictors and criteria. Table 13 shows correla-
tions of the predictors with PSA and SBI rating
dimensions. The only significant correlations
were between the training measures and the
Organizing/Planning rating dimension of the SBI.
Regression analyses, shown in Table 14, were
conducted to assess the relative contribution of
the training measures in predicting each of the
Tating dimensions in the SIDP program. For
tower cab controllers, none of the training per-
formance measures significantly predicted the
PSA scales, but the number of years required to
reach FPL status predicted ratings on beth the
Organizing/Planning and the Direction/Motiva-
tion SBI rating dimensions. Howevez, the per-
centage of variance in the ratings accounted for
by the number of years to reach FPL status was
less than 5%.

DISCUSSION

The purpose of this study was to assess the
contribution of measures of ATCS field training
performance in predicting 1) outcomes in the
multiple hurdles of the Supervisory Identification
and Development Program, 2) ratings made by
peers and supervisors, and 3) ratings of know-
ledges and skills made by a panel during role
plays.

Descriptive statistics were computed and
regression and logistic regression analyses were
conducted for all terminal and en route con-
trollers combined. The Instructor rating in on-
the-job training was related to referral for the
SBI and training status was related to supervisory
selection.

Analyses were also conducted separately for
en foute, terminal radar approach control, and

tower cab air traffic controllers because they
perform different job functions and undergo
different types of training. The measures of field
training performance were found to contribute in
different ways to the prediction of success in
SIDP for the controllers in different options. in
the en route option, instructor catings were
related to referral to the SBI, and the amount of
time required to reach FPL status was related to
supervisory selection. None of the training
measures predicted eligibility for consideration as
a supervisor. For terminal radar approach
controllers, the number of years required to
reach FPL status was related to referral to the
SBI, but none of the training measures predicted
eligibility for consideration or actual <election.
For tower cab controllers, none of the training
measures contributed significantly to the
prediction of status in SIDP. in ail cases, the
logistic regression analyses produced modeis
which predicted that all applicants would be
referred for consideration and no applicants
would be selected. Thus, while statistically
significant results were observed, no meaningful
distinction among candidates was found.

Additional regression analyses were conduc-
ted which included the technical training perfor-
mance measures that were specific to each con-
troller option. For en route and terminal radar
approach controllers, the results suggested that
candidates’ technical proficiency demonstrated in
training was related to subsequent peer and
supervisor ratings of technical competence. The
Measures of technical proficiency contributed to
the prediction of ratings made by peers and
supervisors who were aware of the candidates’
technical skills. On the other hand, those who
rated candidates during the SBI did not rate nor
were they familiar with the candidates’ technical
proficiency. Correspondingly, the measures of
candidates’ technical training performance were
generally not significantly or highly predictive of
their SBI ratings. This outcome is as expected;
the purpose of the SBI is to reinforce selection
on factors other than technical competence alone.

Furthermore, in examining more closely the
correlations between measures of training perfor-
mance and PSA ratings, the measures of the
amount of time required to complete certain


making up the “area of specialization." These
Area Supervisors ensure that available controller
staffing is optimally deployed (Federal Aviation
Administration, 1589). Technical competence is
required of these Area Supervisors to interpret
sector and area activity, and to render assistance
to the controller teams as needed. An analogous
Situation can be found among the supervisors on
the tower lab and Terminal Radar Approach
Control (TRACON) facilities. These supervisors
are selected from the controller workforce thr-
ough the SIDP.

The Supervisory Identification
and Development Program

The SIDP was designed to emphasize fac-
tors such as communication and leadership, in
addition to technical capability, as superviscty
selection criteria. As described by Myers (1992),
SIDP utilized a 2-stage procedure to select
supervisors. Applicants were first rated by peers
and supervisors on a number of dimensions in
the Peer/Supervisory Assessment (PSA) stage of
SIDP. Candidates scoring above the cutoff on a
composite of PSA ratings then participated in a
Skill-Based. Interview (SBI). Candidates who
were successful in the SBI were then placed on
an “Eligible for Further Consideration” (EFC)
list from which supervisors were actually selected
as openings became available.

Because it may be important that supervi-
sors in the ATCS occupaticn demonstrate tech-
nical competence, we hypothesized that success-
ful SIDP candidates, in general, would demon-
strate better technical performance as early in
their careers as the FAA Academy initial qualifi-
cation courses. We reasoned that (a) if persons
doing well in initial training did well in field
training, as the available data suggested, and that
(b) if persons doing well as FPL controllers, then
(c) such persons were more likely to be selected
into first-line supervisory positions through
SIDP. We tested this reasoning by investigating
the following 3 research questions:

(1) How did scores in the initial qualification
training courses correlate with subsequent
SIDP PSA measures that included a rating
of technical competence?

17

(2) Did SIDP applicents, selected through the
SIDP program, de.sonstrate better technical
skills in the initial qualification courses than
unsuccessful SIDP applicants?

(3) Did Academy initial qualification course
scores predict future selection as a first-level
supervisor in the SIDP program?

METHOD
Subjects

ATCSs who entered the FAA Academy in
either the En Route or Terminal screening pro-
grams between August 1981 and September 1985
and applied to the SIDP between 1985 and 1988
were selected for these studies. Some applicants
changed options after they passed the FAA
Academy initial qualifications programs. For
purposes of this analysis, only those applicants
who completed the Academy and were still
working in their original option were included.
Of the 1,128 applicants meeting these sampling
requirements, 468 were in the En Route option,
while 660 were in the Terminal option.

All of the controllers in this sample entered
on duty with the FAA after the illegal 1981
strike and before the FAA Academy program
changed in October, 1985. The sample was
predominantly male (85%) and nonminority
(93%). The modal formal education completed
was “some college" (48%), with about a third
(33%) having completed a baccalaureate degree.
About a third (4%) had at least some prior
aviation-related experience prior to joining the
agency.

Three groups of applicants were compared,
based upon success in the SIDP selection pro-
cess: (1) candidates who were successful on the
PSA and referred to the SBI (Referred versus
Not referred); (2) based upon the SBI, candi-
dates were deemed “eligible for consideration"
(EFC) or not (EFC versus Not EFC); and (3)
candidates selected as supervisors from the EFC
list or not (Selected versus Not selected).

Three-quarters of the sample (N = 844) were
teferred to the Skill-Based Interview step in the
SIDP. About haif (55%, N = 615) of the total


TABLE 2

FAA Academy ATCS initial qualifications training measures

Phase Weight Measure Description
Academics 10% Block test average (BA) 4 multiple-choice instructional block tests
and map test
Comprehensive Phase Test comprehensive multiple-choice test over all
(CPT) academic topics
Laboratory 65% Average technical score average technical performance assessments
(AVTA) on best 5 out of 6 graded laboratory
problems
Average instructor score average instructor assessment of potential
(AVIA) on best 5 out of 6 graded laboratory
problems
Average laboratory score erage overall score on best 5 out of 6
(AVL5) graded | laboratory problems
Final examination 3% Controller Skills Test (CST)~ comprehensive, multiple-choice examination
Final score Final composite score (COMP) 70% required to pass

TABLE 3

Correlations between FAA Academy performance measures and PSA ratings by ATC option

PSA Rating Categories
Measure Communication Leadership Interpersonal Technical Composite
En Route
BA 061 .092* 080 -104* 089
CPT 059 08S 050 081 O73
AVLS 034 -023 -.032 -106* 032
AVTA 028 -020 ~.034 .097* 027
AVIA 046 -031 -.013 -107* 042
cSsT -106* A1L8* 089 .143* -120*
COMP -068 065 -006 -144* 072
Terminal
BA -103* .081* 065 .039 .079*
CPT -151* -128* 127+ .084* 132*
AVLS 040 044 ~.004 -046 .033
AVTA 056 070 016 063 055
AVIA 007 027 -.030 018 005
CST O71 -093* 043 .081* .079*
COMP -080* -090* 028 -082* O75
*p< 0

23


Because the PSA contained an evaluation of
the applicant’s technical abilities, an analysis was
conducted to determine the extent of the relation-
ship between Academy measures and PSA rat-
ings. Table 3 presents the correlations among the
SIDP PSA measures and Academy scores for the
En Route and Terminal options. Results of these
analyses revealed that there were small but
statistically significant correlations between all of
the Academy measures and the PSA Technical
rating, except for the CPT, for the En Route
option. In the Terminal option, however, the
hypothesized correlations between the PSA
Technical ratings and laboratory measures were
not found.

Research question 2: Academy performance -
SIDP outcomes

The purpose of research question 2 was to
compare Academy performance of successful
SIDP applicants to unsuccessful applicants at
each stage of the SIDP selection process. We
hypothesized that successful SIDP applicants
demonstrated better technical performance as
early in their careers as the Academy initial
qualification training courses. Table 4 presents
descriptive statistics for the SIDP applicants’
Academy performance scores by outcome at each
stage of SIDP. Results of MANOVAs utilizing
the Academy variables as dependent variables
revealed no statistically significant differences
between groups for (1) referral to SBI for either
En Route or Terminal, (2) eligibility for con-
sideration for Terminal only, and (3) selection
for either En Route or Terminal supervisor. A
MANOVA comparing the En Route EFC and Not
EFC groups was significant (F = 2.73, p <
.01). Univariate analyses of variance (df = 1,
349 in each analysis) demonstrated significant
differences between EFC and Not EFC groups on
the following Academy variables: AVLS (F =
7.33, p < .01); AVIA (F = 12.96, p < .001);
AVIA (F = 5.61, p < .05); and COMP (F =
4.83, p < .05). The mean score for each of
these variables was lower for the EFC group than
for the Not EFC group. The hypothesis that
successful SIDP candidates had demonstrated
better Academy performance than their unsuc-
cessful SIDP counterparts could not be sup-
ported.

Research question 3: Academy performance
and the odds for success in SIDP

As ATCS initial qualification couzse scores
predicted technical job performance, and since
that technical competence was required of con-
troller first-line supervisors, we hypothesized that
better performance in the ATCS Academy initial
qualification courses improved the odds for
referral to the Skill-Based Interview, for making
the Eligible for Consideration list, and for seiec-
tion as a supervisor. The independent variables,
representing Academy performance, were: acade-
mics block test average (BA); academics com-
prehensive phase test (CPT); average of the 6
graded laboratory problem technical performance
assessments (AVTA); average of the 6 graded
laboratory problem instructor assessments of
potential (AVIA); and score on the final Con-
troller Skills Test (CST). These independent
variables were logistically regressed on the
categorical outcomes at each stage of SIDP.
Results of the logistic regression, used to test this
hypothesis, are presented below.

Referral to SBI. The probebility of being
referred to the SBI was estimated using the SPSS
logistic regression procedure (Norusis, 19°C), the
independent variables were entered into the
equation, and model goodness-of-fit evaluated.
The analysis indicated that previous performance
in technical training did not predict referral to
SBI in either the En Route or Terminal options.
Assessment of the goodness-of-fit using -2 times
the log likelihood (-2 LL) of the model indicated
a poor fit to the data (En Route: -2 LL
x°(467)=526.3, p < .05; Terminal: -2 LL
x°(658) = 737.0, p < .01), where the null
hypothesis was that the model fit the data.
Inspection of the classification table resulting
from the model supported the conclusion of a
poor modei fit. While the overall correct
classification rate of 75.0% and 74.7%, for En
Route and Terminai, respectively, appeared
acceptable, all but 2 of the Terminal cases not
actually referred to the SBI were predicted as
referrals. In other words, the model predicted
that all, except 2, cases would have been referred
to the SBI. Thus, performance in the ATCS
initial qualification courses did not provide useful
information about characteristics predicting


training phases predicted ratings on scales pres-
umably unrelated to technical skills—leadership,
communication, and interpersonal skills. This
finding suggests that perceptions of technical
proficiency may bias how peers and supervisors
interpret other aspects of employees’ perfor-
mance, and subsequently influence their inter-
pretation of supervisory potential. On the other
hand, the rating dimensions may tap overlapping
characteristics; for example, peers and
supervisors may interpret leadership to include
technical components, thus resulting in significant
correlations between training performance and
tatings made on other dimensions.

Moreover, measures of technical proficiency
and instructor ratings both contributed to the
prediction of ratings made on the PSA Technical
scale, although the instructor rating did not
contribute significantly to regression models
predicting ratings made on any of the other PSA
scales. This result might suggest that some factor
other than performance alone contributes to the
assessment of technical proficiency, but such a
speculation is not confirmed by observing
particularly high correlations between instructor
fatings and other PSA rating scales that might
measure this other factor (such as communication
or interpersonal skills).

Finally, similar results were not observed for
tower cab controllers. One reason might be that
insufficient numbers of candidates had trained at
tower cab facilities as their first facility. Also, it
is possible that the measures of training
performance do not describe technical proficiency
for the tower cab controller as well as they do
for other types of controllers who spend longer
periods of time in training.

In summary, available measures of field
training performance, though hindered by mea-
surement problems, are somewhat predictive of
both ratings of future potential for success as a
first-line supervisor and of successfully
completing the SIDP program, a prerequisite for
becoming a first-line supervisor. In one sense,
the result is somewhat predictable, because at
Jeast minimal technical competence is a
requirement for a first-line supervisor (however,
see Siusher, Van Dyke, & Rose, 1972).

32

Although Della Rocco & Broach, (1991)
suggested that technical proficiency measured in
the FAA Academy was not predictive of SIDP
outcomes, the training performance measures
used here occurred in a setting that resembled the
job more closely than did the ATC screen
program, and the measures were obtained more
recently than were the screen performance
measures.

On the other hand, the results are somewhat
unexpected, because a) the training performance
measures are fairly global and do not describe
specific strengths or weaknesses in performance,
b) instructor ratings were usually made by
personnel untrained in the use of the rating scales
{and thus, were very likely tainted by a number
of rater biases), c) the measures were obtained
during training and refiect skill learning rather
than job performance, and d) a number of other
factors besides technical success are assessed in
determining success in the SIDP program. At the
same time, it must be remembered that none of
the measures of training performance accounted
for more than 10% of the variance in ratings of
supervisory potential made for candidates to the
SIDP program. At best, these measures might be
added to regression models containing other
variables to provide additional contribution to the
prediction of SIDP ratings.

Tt is not suggested that the available measures
of field training performance sufficiently describe
controller performance. A future research study
is planned to identify and develop better mea-
sares of performance during training and on the
job. When those measures are available, it will
be interesting to return to the question of the
relationship between technical performance and
supervisory success and determine whether the
relationships found in this study are still rele-
vant.


b aNd

= = = a = » O06 4
= re] = s a * O22 M 'E
2 ] = a ro) @ oge N ‘2
= r) = a m 8% O9€ N ‘b
‘ANOO HLYON LSAM HiNnOS 1SV3
(3) (a) (9) (a) (7) SaTanvx3
HLYON . O09€ 40 3 Jo N
1ISSM . 022 40 » JO M
HLNOS . osi 40 2 40 s
isva . 06 40 ¢ 40 q
ONIdVaH 53350540 MOUEV gaits
Widsid Viva


MCAT scores to a battery of cognitive tests,
concluded that the MCAT measures “both per-
ceptual and cognitive reasoning abilities." This
result was consistent with the research findings as
reported in Sells, Dailey, and Pickrel (1984).

The ABSR consists of a series of figuzes or
ietters that illustrate principles of logic. The
objective is to select from several alternatives the
one that correctly completes a sequence. The
ABSR measures aspects of logical reasoning and
Spatial-perceptual ability (see Figure 2). The
OKT, a test of job knowledge, consists of ques-
tions about air traffic control phraseology and
procedures, air navigation, and aviation weather.
A transformed score, referred to as the Trans-
muted Composite (TMC), is derived by weight-
ing the MCAT and ABSR to produce a distribu-
tion with a mean of 70 and a maximum score of
100. The final OPM rating includes the addition
of extra points for the applicant’s OKT score and
veteran’s preference points. OPM selection
battery performance measures for this study
included the raw scores for the MCAT, ABSR,
and OKT and the TMC.

Dial Reading and Directional Heading Tests.
The DRT and DHT are 2 experimental tests that
were completed by a small sample of the SIDP
applicant group. Tre DRT is a brief timed test in
which individuals are asked to identify and
correctly inte:pret seven different instrument
dials for a series of questions (see Figure 3). The
DRT was developed in the 1950’s for the US Air
Force to select candidates for undergraduate pilot
training. Research by Boone (1979), Marshall-
Mies and Colmen (1976) and Schroeder, Dollar
and Nye (1990) has demonstrated the utility of
the DRT in predicting success of ATCS trainees.
The DHT is a speeded paper and pencil test of
spatial ability. Each item is comprised of 3 bits
cf information that reflect the cardinal points on
a compass (e.g., the letter "W", a symbol "< ",
and the notation "270", each denote “West”, see
Figure 4). Other letters, symbols, and notations
refer to the other cardinal points on a compass
(North, East or South). In the first of 2 parts of
the DHT, the individual is asked to determine the
direction indicated (if all information is consis-
tent), or indicate that the information is inconsis-
tent. In the second part of the test, the direction

can be determined if 2 or more of the 3 types of
information presented agree; the direction cannot
be determined when ail 3 types of information
disagree. Research has also demonstrated that the
DHT can be used successfully to select personnel
for ATCS training (Cobb & Matthews, 1972;
Boone, 1979; Schroeder, Dollar & Nye, 1990).

SIDP Performance. The SIDP process was
described in the introduction by Mvers (1992).
PSA ratings used in this investigation inciude: the
composite (COMP); communication (COMM);
interpersonal skills (INTR); leadership (LEAD);
and technical competence (TECH). Rating results
from the Skill-Based Interview were summed to
create an overall score (SBI TOTAL).

Analyses. Means and standard deviations for
the OPM selection battery were obtained for the
initial ATCS applicant group, entrants to the
FAA ATCS Screen program, those who success-
fully completed the ATCS Screen program, SIDP
applicants, SIDP candidates placed or the list of
eligibles, and eligibles selected to fill supervisory
positions. Separate analyses were completed for
ATCSs in the terminal and en route options.

Correlations among OPM selection tests,
DHT, and DRT with the PSA and SBI com-
ponents of the SIDP were computed. The ana~
lyses were conducted separately for the 2 op-
tions.

RESULTS

Means and standard deviations of scores on
each component of the OPM test battery for en
toute and terminal option ATCSs are presented
in Table 1. The effects of restriction in range
that occur as part of the selection process «ze
evident in the higher MCAT scores of Academy
entrants who subsequently enter the en route
option (En Route M=90.0, Terminal M=88.5
versus M=69.7 for all applicants) and the cor-
responding smalier standard deviations (SD of
7.1 in en route; 7.3 terminal versus 16.1 for
applicants). These differences were also mirrored
in the ABSR scores tor both options. The lack of
evidence of any restriction in range for the OKT
is due, in part, to the small number of applicants
who actually have an ATC background. With the


significantly higher instructor ratings than those
not referred. (F(1,456) = 14.3, p <.001), and
used significantly fewer OJT hours to complete
training than did those not referred (F(1,453) =
4.49, p < .04). No differences in field training
performance as a function of eligibility for
consideration were observed. Those selected as
supervisors spent significantly fewer years in
training (F(1,239) = 7.65, p < .01), and
significantly fewer days in OJT, (F(1,243) =
6.34, p < .G2), than did those not selected.

Logistic regression analyses were conducted
to assess the relative contribution of the training
Measures in predicting status in each phase of
SIDP. A model was developed to predict SIDP
status, using the following as independent vari-
ables: the number of hours and calendar days
used to complete OJT on the first 2 sectors,
instructor ratings for OJT phases, and the
amount of time (in years) to reach FPL status. A
model containing mean instructor rating sig-
nificantly predicted referral to the SBI ((1) =
16.5, p < .001) while the other variables con-
tributed nothing more to the prediction of che
model (x7(3) = 1.95, p > .58). None of the
variables contributed significantly to a model
predicting eligibility for consideration (x2(4) =
6.61, p > .15). A model containing only the
time to reach FPL status significantly predicted
supervisory selection (7(1) = 7.20, p < .01),
while the other variables contributed nothing to
the prediction of the model (x’(3) = 4.16, p >
-24).

It must be noted that the goodness of fit of
the model is somewhat questionable. The -2 LL
Statistic, indicating how well the model fits the
data, was significantly different from 1 for the
prediction of SBI referral status -2 LL =
488.15, p < .04, df = 435) but the goodness
of fit statistics for the same model did not lead
to the rejection of the model (Goodness of fit =
436.0, p > .47, df = 435.) The prediction of
eligibility for consideration by the logistic
regression model resulted in a classification of
all but 2 applicants as “eligible.” Although both
goodness of fit statistics for the model predicting
supervisory selection suggested that the model fit
the data, the prediction of selection by the model
resulted in a classification of all candidates as

29

“not selected.” Thus, while the model successful-
ly predicted EFC and supervisory selection in a
statistical sense, the classifications predicted by
the model are of questionable utility.

Question 3: For en route controllers, in what
ways are the measures of training per-
formance related to the rating dimensions used
to determine success in SIDP?

Table 4 shows descriptive statistics for the
predictors and rating dimensions for en route
controllers, and Table 5 shows correlations
between the predictors and the SIDP rating
dimensions. Several of the measures of training
performance had statistically significant, though
somewhat low, correlations with the PSA ratings,
and had virtually no relationship with ratings
from the SBI.

Regression analyses, with results shown in
Table 6, were conducted to assess the relative
contribution of the training measures in predict-
ing each of the rating dimensions in the SIDP
program. While the training performance mea-
sures were significantly related to ratings made
on PSA dimensions, very little relationship was
observed between training measures and ratings
made on the SBI. OFT hours and the number of
days in OJT were significantly related to the
composite score, and the Communications,
Interpersonal and Leadership rating dimensions
of the PSA. A model containing both OJT time
and instructor rating predicted the PSA Technical
tating dimension. However, most of the SBI
rating dimensions were not predicted by
measures of training performance. The number
of years to reach FPL status had a modest
relationship with ratings on Knowledge of
Supervisory Role Performance; OJT time had a
modest relationship with the ratings on
Decisiveness. Training performance measures
accounted for 10% cr less of the variance in the
PSA and SBI ratings.

Question 4: For terminal radar approach
controllers, do any of the specific measures of
training performance predict success in SIDP?

The measures of technical training perfor-
mance relevant to the analyses dealing with



on each sector before radar training begins; at
other facilities, both radar associate and radar
treining are provided for a specific sector before
training begins on the next sector. En route
controifers must complete up to 9 phases of field
training, depending on the type of training
provided by the facility.

In the terminal option, there are 2 different
types of controllers. At approach control facil-
ities, the terminal controllers use radar equipment
to separate aircraft converging on or departing
from an airport, although these aircraft are
slower and closer together than they are in the en
route environment. Approach controllers may
work in pairs but often work alone to manage the
aircraft within their radar position (equivalent to
an en route sector). Thus, training for terminal
approach controllers generally involves the
combined duties of a radar controller, instead of
splitting out the functions into several training
phases, as is seen in the en route environment.
Also, approach controllers can be required to
train to journeyman level in up to 10 sectors
(called “positions” in the terminal environment)
in comparison to a maximum of 8 sectors for en
route controllers. Approach controllers take at
least 2 phases of field training (flight data and
radar, asd oonradar training is optional) but may
take up to 6 phases if their facility is combined
with a tower cab.

Tower cab controllers control traffic landing
or taking off from an airport. At tower cab
facilities, controllers perform duties associated
with different independent functions, e.g.,
{entering and updating flight data, delivering
clearances related to the flight to pilots who have
not yet entered the taxiway, directing air traffic
moving along the taxiways back and forth from
the runway, and directing aircraft to take off and
land). Some of the functions may be combined,
but the separate functions are not usually
performed as a team. Thus, training is conducted
on 1 function at a time. Tower cab controllers
complete 4 phases of field training.

Available Measures of Field
Training Performance

Several types of information on performance
are obtained for each phase of air traffic control
field training: the start and completion dates, the
number of hours used to complete OJT, and the
gtade (Pass, Fail, or Withdraw). A rating of
controller potential, measured on a 1 to 6 scale,
is made by an instructor or supervisor who most
frequently observed the student during that phase.
This information can be compiled to derive
Measures of training performance, such as the
amount of time (in years) required to reach
journeyman or full performance level (FPL)
Status, mean instructor ratings computed across
certain training phases, time (in days) to
complete OJT in certain training phases, and total
number of OJT hours required to complete those
phases. Total OJT hours and days generally vary
by facility type (Manning, Della Rocco &
Bryant, 1989). For example, en route controllers,
average 2.7 years to reach FPL status, while
terminal approach controllers average 1.7 years;
average training completion time for tower cab
controllers is .7 years.

Although these assessments of training per~
formance are available for most students, it must
be understood that a number of outside factors
(besides aptitude and technical proficiency) may
affect the accuracy of their measurement. For
example, time to reach FPL status may be
affected by delays in training caused by the need
to use the controller in an operational position,
the number of other students undergoing OJT on
the same airspace, and/or the availability of the
training laboratory. The number of OJT hours
used may be affected by insufficient exposure to
different types of traffic during training. The
subjective rating of student potential could be
affected by a number of rating biases familiar to
psychologists (e.g., leniency, central tendency,
severity, halo effect, contrast and similarity
errors). Withdrawal from raining before
completion usually occurs because of training


face interview, the candidate is asked 8 questions
to assess Organizational Knowledge and Know-
ledge of the Supervisory Role. The role-plays
are designed to simulate realistic situations en-
countered as a supervisor. Based on their perfor-
mance in the scenarios, applicants are rated on 7
competencies (Problem Solving and Analytical
Ability, Judgment, Decisiveness, Organizing and
Pianning Ability, Interpersonal skill, Communi-
cation Skill, and Direction and Motivation) by a
trained panel of 3 interviewers and receive verbal
and written feedback following the interview. A
description of the competencies are contained in
Appendix A.

There are 3 possible direct outcomes from
the SBi: referral to (a) self-development, (b) the
Candidate Review Board, or (c) the Eligible for
Consideration list. Those who demonstrate
ineffective performance in the SBI are counseled
on their strengths and weaknesses and recommen-
ded for self-development prier to reapplying to
the SIDP. Those who perform at a moderate
level are referred to a Candidate Review Board.
The board determines whether the individual will
be placed in formal development for skill remed-
iation or will be recommended for self-develop-
ment activities before reapplying to a future
announcement of the SIDP. Those who are
assigned to formal development are provided
resources for receiving training and/or additional
experiences (e.g., details or “shadowing” assign-
ments) that will remediate skill deficiencies and
result in eligibility for supervisory positions.
Applicants who perform at a high level on each
of the competencies are placed on an Eligible for
Corsideration list and are referred to position
vacancies.

Individuals who are on the eligible for con-
sideration list are polled for their interest in
applying for vacant supervisory positions. Once
a list of eligible and interested bidders is es-
tabiished, the list is sent to the selecting officials
along with a packet of information containing the
PSA and SBI results and work history informa-
tion (awards, training, and job experience) for
each individual referred to the vacancy.

As demonstrated in the description of the
ATCS selection and training program, individual

res

advancement and recognition in nonsupervisory
positions is based on the achievement of a high
level of technical competence. However, for
promotion to a first-line supervisor position,
ATCSs must demonstrate not only technical
competence but also skills in areas such as
decision-making, communication, anc leadership.
Several questions arise in the selection of super-
visors from a technical workforce: Given that
technical competence is required to perform the
first-line supervisor job, does prior technical
training and journeyman performance relate to
“promotability” or actual selection for a super-
visor position? What characteristics distinguish
those who are selected from the eligible pool
from those that are not selected? These questions
are addressed in the following papers, which
examine ATCS entry level, screening, and
training performance in relation to SIDP perfor-
mance.



TESTING BY OFFICE OF PERSONNEL MANAGEMENT

Ad
ATTEND ACADEMY SCREEN
i
v
ASSIGNED TO FIELD FACILITY
>
TERMINAL EN ROUTE
v v
{
Vv v
RADAR TRAINING FULL PERFORMANCE
AT ACADEMY LEVEL
v
Our
v
FULL PERFORMANCE
LEVEL

AIR TRAFFIC CONTROLLER SCREENING AND TRAINING

Figure !

APPLICATION TO PROGRAM
v
PEER AND SUPERVISORY ASSESSMENT
|

Vv
SKILL-BASED INTERVIEW

<< |.oo=TM::__ >

ELIGIBLE FOR v SELF
CONSIDERATION CANDIDATE DEVELOPMENT
FOR PROMOTION REVIEW BOARD

FORMAL DEVELOPMENT SELF DEVELOPMENT
ELIGIBLE FOR SELF

CONSIDERATION DEVELOPMENT

AIR TRAFFIC SUPERVISORY IDENTIFICATION
AND DEVELOPMENT PROGRAM

Figure 2



AIR TRAFFIC CONTROL SPECIALIST TECHNICAL COMPETENCE
IN INITIAL TRAINING AND SELECTION AS A FIRST-LINE SUPERVISOR

Pamela S. Della Rocco, M.A. and Dana Broach, Ph.D.

A first-line supervisor holds a key spot in an
organization, serving as the main point of contact
between the rank-and-file and management and as
the person responsible for the day-to-day or-
ganizational performance. The selection of first-
line supervisors was considered essen-tially
settled as a result of many studies between 1947
and 1965 (Dooher & Marting, 1957; Raza,
1987). Corporations generally selected first-line
supervisors on the basis of their technical skills,
rather than managerial skills (Northrup, Cowin,
Vanden Plas, Fulmer, Bolick, Bellace, & Rosen-
zweig, 1978; Patton, 1974).

However, evidence from indirect sources
suggests that such selection procedures for first
line supervisors may be suboptimal. For ex-
ample, Raza (1987) noted that surveys of first
line supervisors suggested a need for training in
communication and management skills. Myers
(1990) reached much the same conclusions
through an analysis of the FAA biennial Job
Satisfaction Survey and Survey-Feedback-Action
Program. Phillips (1985) argued that while
technical skills and experience may be important,
they are not necessarily related to the skills and
abilities required of a successful supervisor.
Given these two disparate lines of research, we
askeé what relationship prior technical perfor-
mance in training, for example, had to promot-
ability and ultimate selection for supervisory
positions. To answer that question, we inves-
tigated the relationship of performance in initial,
technical air traffic control specialist (ATCS)
training to selection as a first-line air traffic
control supervisor through the Federal Aviation
Administration Supervisory Identification and
Development Program (SIDP).

Air Traffic Control Specialist Initial
Qualification Training

Since 1976, candidates seiected for the
occupation of Air Traffic Control Specialist
(ATCS) with the FAA, in the En Route and
Terminal options, were required to complete a 2

45

stage selection process. The first stage consisted
of a battery of written tests and was administered
by the U.S. Office of Personnel Manarement
(OPM, tormerly the Civil Service Commission)
prio- to hiring by the FAA. The second stage of
the ATCS selection process was a performance-
based pass/fail initial training course administered
at the FAA Academy after FAA hire. This study
focuses on ATCS performance in the Academy
initial qualification courses as a potential predic-
tor of selection as a first line supervisor.

ATCSs work in 1 of 3 options: En Route,
Terminal, and Flight Service Station. En Route
and Terminal controllers ensure the separation
of aircraft by using information about the speed,
direction, and altitude of aircraft to formulate
clearances and communicate them to pilots.
Clearances are sets of instructions for pilots,
designed to ensure the safe, orderly, and ex~-
peditious flow of air traffic. En Route controllers
ensure the separation of aircraft traveling be-
tween airports. There are 2 types of terminal
controllers. Terminal radar approach and depar-
ture controllers use radar to separate aircraft
converging on or departing from an airport.
Tower cab controllers control traffic landing or
taking off from the airport, most often without
the direct use of radar. First-line En Route and
Terminal supervisors oversee and coordinate the
actions of controller teams responsible for spec-
ific sectors, or segments of contiguous airspace.
Flight Service Station specialists, on the other
hand, provide other services to pilots, such as
weather briefings, filing flight plans, and helping
to locate lost aircraft. Flight Service Station
specialists do not issue control instructions to
aircraft; rather, they relay En Route or Terminal
controller clearances, as appropriate. Given the
fundamental difference in the options, we focused
on En Route and Terminal ATCSs and their
supervisors, as they are responsible for issuing
clearances to aircraft, and excluded Flight Ser-
vice Station ATCSs from this analysis.

The FAA Academy initial qualification


COGNITIVE INDICATORS OF ATCS TECHNICAL ABILITY AND
PERFORMANCE IN A SUPERVISORY SELECTION PROGRAM

David J. Schroeder, Ph.D.

Historically, first leve' supervisors have been
selected with respect to their technical com-
petence and their performance on the job they
will supervise. Raza (1987), in a comprehensive
review of the literature concerning the selection
of first level supervisors, concludes that while
supervisors, researchers, and human resource
personnel have for a number of years felt that
interpersonal skills are critically important factors
in the performance of first-level supervisors,
current selection procedures for those positions
have, by and |? ~e, continued to emphasize
technical competence. Furthermore, Raza (1987)
provides convincing evidence that current pro-
cedures are less than optimal. The Supervisory
Identification and Development Program (SIDP),
developed by the FAA, is designed to incor-
porate information other than technical perfor-
mance of .he applicant in the selection process
for supervisory Air Traffic Control Specialists
(ATCSs).

Applicants to the ATCS job must first com-
plete a 2-stage selection process; a hatterv of
selection tests admirusterca oy the Office of
Personnel Management (OPM) and a 9 week,
performance-based Screen program at the FAA
Academy in Oklahoma City. This investigation
was developed to determine the relationship
between entry level information on applicants’
cognitive aptitudes for performing ATCS work,
as measured by tests in the OPM selection bat-
tery, and subsequent selection as a potentia!
supervisor through the SIDP. Specifically, the
interest was in determining if scores on the initial
selection tests predict technical performance
tatings received by the SIDP applicants. This
analysis was undertaken in light of the previous
research (Raza, 1987) that suggested that existing
procedures for supervisory positions stil! tend to
emphasize technical competence. If that remains
true for the SIDP process, entry leve! measures
of technical aptitude may well predict subsequent
SIDP performance.

METHOD
Subjects

The OPM selection battery is administered to
all applicants to determine hiring eligibility.
OPM scores were entered into the analytic data
base for 2,493 ATCSs who applied to SIDP
between 1985 and 1989. Of this group, data were
available for 1,215 who entered into the en route
option and 1,084 in the termina! option. This
division was based on the nature of the Academy
Screen program (terminal or en route) and the
type of facility to which the ATCSs were subse-
quently assigned following completion of the
Academy Screen. The 2 types of facilities do
involve significant differences in the overall
compiexity and type of air traffic they control. A
small sample of en route and terminal Academy
entrants also completed 2 experimental tests - the
Directional Headings Test DHT: en route
N=238, terminal N= 136 and Dial Reading Test
(RT: en route N= 139, terminal N= 134).

Measures

OPM Seiection Battery. The current OPM
selection battery was implemented in 1981 (see
Sells, Dailey and Pickrel, 1984 for information
concerning the validation of the test battery) and
is comprised of the Multiplex Controller Aptitude
Test (MCAT), the Abstract Reasoning Test
(ABSR), and the Occupational Knowledge Test
(OKT). The MCAT is a speeded paper and
pencil test that presents 2 simulated cortroller
work sample. A map containing several aircraft
on various air routes is presented, along with
tabular data concerning the aircrafts’ altitude,
speed, and route for each question. Aircraft
positions and interpretation of the tabular data are
required to resolve questions concerning tizie and
distance and possible conflicts between aircraft
(see Figure 1). Harris (1986), in a study relating


supported. Although the average of most of the
Academy scores were slightly higher for the
successful candidates, the difference was not
significant. In fact, the successful candidates
deemed “eligible for furtier consideration" had
Statistically significantly lower average scores
than the unsuccessful candidates in the En Route
option. No significant differences were found
between any of the SIDP groups in the Terminal
option.

The final research question investigated the
extent to which Academy scores predicted selec-
tion in SIDP using logistic regression. The an-
alyses, based on a relatively large sample of
controllers, indicated that performance in initial
technical training was, in fact, not predictive of
referral to the initial stages of a Supervisory
Identification and Development Program nor of
selection as a supervisor.

On one hand, these data lend support to the
contention by Phillips (1985) that technical skills
and experience may be unrelated to the skills and
abilities required of supervisors. On the other
hand, it may be argued that the initial technical
training is so distant, both temporally and con-
ceptually, from the supervisory selection process
as to be meaningless. Indicators of technical
performance measured closer in time to con-
sideration for SIDP, such as performance in field
training, may be more indicative of managerial
potential. Another alternative interpretation is
that the level of homogenous technical com-
petence within the workforce is assured by the
nature of the upfront screening process repre-
sented by the ATCS initial qualification courses.
Thus, other factors, such as communication
skills, knowledge of agency programs, may be
more predictive of successful completion of the
multiple hurdies embedded within SIDP.

REFERENCES

Della Rocco, P. S., Manning, C. A., & Wing,
H. (1990). Selection of controllers for auto-
mated systems: Applications from current
research, Washington, D.C.: FAA Office of
Aviation Medicine Report No. DOT/FAA/-
AM-90/13.

21

Federal Aviation Administration. (1989,
February). Facility operation and admin-
istration. (FAA Order 7210.31). Washing-
ton, D.C.: Federal Aviation Administration
Air Traffic Service.

Hosmer, D. W. Jr., & Lemeshow, S. (1989).
Applied logistic regression. New York:
McGraw-Hill.

Manning, C. A., Della Rocco, P. S., & Bryant,
K. LD. (1989). Prediction of success in FAA
air traffic control field training as a function
of selection and screening test performance.
Washington, D.C.: FAA Office of Aviation
Medicine Report No. DOT/FAA/AM-89/6.

Dooher, M. 3., & Marting, E. (Eds.). (1957).
Selection of managemert personnel. (Vols. 1
& 2). New York: American Management
Association.

Myers, J. G. (1990). Management assessment:
Implications for development and training.
Washington, D.C.: FAA Office of Aviation
Medicine Report No. DOT/FAA/AM-90/2.

Myers, J5.G. (1992). An overview of the air
traffic control specialist and first-line super-
visor selection systems. In J. G. Myers
(Ed.), A longitudinal examination of ap-
plicants to the Air Traffic Supervisory Iden-
tification and Development Program. Wash-
ington, D.C.: FAA Office of Aviation Med-
icine Report No. DOT/FAA/AM-92/16.

Northrup, H. R., Cowin, R. M., Vanden Plas,
L. G., Fulmer, W. E., Bolick, R. E. Jr.,
Bellace, J. R., & Rosenzweig, A. H. (19-
78). The objective selection of supervisors:
A study of informal industry pactices and two
models for improved supervisor selection.
(Manpower and Human Resources Studies
No. 8). Philadelphia: University of Pennsyl-
vania, Wharton School of Business, In-
dustrial Research Unit.

Norusis, M. J. (1990). SPSS Advanced statistics:
User’s guide. (2d Ed.}. Chicago, IL: SPSS,
Inc.



DOT/FAA/AM-92/16

Office of Aviation Medicine

Washington, D.C. 20591

AD-A251 879
Ce

92 6 15

113

A Longitudinal Examination of |
Applicants to the Air Traffic Control |
Supervisory Identification and |
Development Program |

DTIC |

ELECTE
JUNL 7 1992

Jennifer G. Myers, Editor

Civil Aeromedical Institute
Federal Aviation Administration
Oklahoma City, Oklahoma 73125

April 1992

Final Report

This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.

2

U.S. Department
of Transportation

See arin 92- 15584
WNL


Table 13

Correlations between SiDP rating dimensions
and training measures for Tower Cab controllers

Training Measures

Years to Instructor OT OT

SIDP Ratings FPL Status Rating Hours Days

PSA Ratings
Composite 01 -08 -.00 OL
Comniunication “4 -96 4H -.01
Interpersot -.O1 .07 ‘ 02
Leadership 1 08 -.00 -.00
Technical 05 Al -.05 03

SBI Ratings
Agency Programs 01 01 -.07 -.04
Supervisory Role -.09 13 .03 -.05
Prob Solv/Analytical -15 ll -.09 1
Judgment -.08 07 -.04 -.10
Decisiveness -.14 03 -.09 -.09
Interpersoeal Stil © a B 08 “0e
af ni ~1S A oA -!
Comeauntcation 15 HH -.04 -.06
Direction/Motivation -1T .06 -.12 -.13

p< 01 p< Ol


event, one should be aware that the aptitude
measures included in this study represent only a
limited number of cognitive abilities that are
critical for on-the-job performance of ATCSs. It
could be that a broader range of aptitude mea-
sures would reveal more consistent and higher
correlations with the various ratings and mea-
sures comprising the SIDP.

Given the current measures of aptitudes for
the ATC profession, there was little support for
any relationship between these aptitue.s and
either overall ratings during the Ski. Based
Interview or selection as a supervisor. However,
background knowledge of ATC and pilot pro-
cedures may erhance the chance of an indi-
vidual’s selection as a supervisor for those
entering the terminal option. One possible ex-
planation is that within the terminal option, the
ATCSs need to be more familiar with aircraft
characteristics and associated rules and pro-
cedures due to the nature of the work activities.

Therefore, those who enter the terminal
option with a more extensive experience in the
area and general background knowledge are per-
ceived to be more suited for a supervisory pos-
zion. However, given the limited number of
variables inciuded in this study and the large
number of additional factors that could play an
explanatory role, additional information and
analyses are necessary to fully understand this
difference.

REFERENCES

Boone, 3. O. (1979). Toward the development of
a new selection battery for air traffic control
specialists. Washington, D. C.: FAA Office
of Aviation Medizine Report No. FAA/AM-
79/21.

Cobb, B. B., & Matthews, J. J. (1972). A pro-
posed new test for aptitude screening of air
traffic control applicants. Washingtcn. D.C.:
FAA Office of Aviation Medicine Report No.
FAA/AM-72/18.

Harris, P. A. (1986). A construct validity study
of the Federal Aviation Administration Multi-
plex Controller Aptitude Test. (Unpublished
technical report). Washington, D. C.: U.S.
Office of Personnel Management, Decembez.

Manning, C. A., Della Rocco, P. S., .& Bryani,
K. D. (1989). Prediction of success in FAA
air traffic control field training as a function
of selection and screening test performance.
Washington, D. C.: FAA Office of Aviation
Medicine Report No. DOT/FAA/AM-89/6.

Marshall-Mies, J., & Colmen, J. G. (1976).
Development of recommendations for ATCS
selection tests. Washington, D.C.: Education
and Public Affairs.

Myers, 3. G. (1992). An overview of the air traf-
fic control specialist and first-line supervisor
selection systems. In J. G. Myers (Ed.), A
longitudinal examination of epplicants to the
Air Traffic Supervisory Identification end
Development Program. Washington, D.C.:
FAA Office of Aviation Medicine Technical
Report No. DOT/FAA/AM-92/16.

Raza, S. M. (1987). Personality characteristics
of effective first line supervisors. Uapublished
doctoral dissertation, University of Tulsa.

Schroeder, D. J., Dollarz, C. S., & Nye, L. G.
(1990). Correlates of two experimental tests
with performance in the FAA Academy air
traffic control nonradar screen program,
Washington, D. C.: FAA Office of Aviation
Medicine Report No. DOT/FAA/AM-90/8.

Sells, S. B., Dailey, J .T., & Pickrel, E. W.
(Eds.). (1984). Selection of air traffic con-
trolle-s. Washington, D.C.: FAA Office of
Aviation Medicine Report No. FAA/AM-
84/2


exception ut a lower average MCAT score for
those selected as supervisors in the terminal
option (88.2 versus 90.2 for SIDP eligibles-but
not selected), there was little difference in
MCAT scores for the different SIDP groups or
for those selected as supervisors. Supervisors
selected in the terminal option had slightly lower
average ABSR scores than eligibles. However
within the en route option, those selected had
slightly higher ABSR scores (41.7 versus 40.6)

The largest difference between the eligible
and selected supervisor groups was evident in the
OKT scores. Of those initially entering the en
Toute option, the average scores for supervisors
(M = 33.3) was statistically below (F(1,466) =
7.4, p<.01) that of the ATCSs who completed
Academy training and entered the same option
but were not yet selected (M = 38.1). In con-
trast, within those entering the terminal option,
eligibles who were selected exhibited higher
average OKT scores (M = 43.5) than those who
graduated from the Academy, applied to become
@ supervisor, or were placed on the list of eli-
gibles (M = 39.3, F (1,459) = 7.6, p<.01).
The present anzlysis suggests that individuals
who enter the terminal option with a more exten-
Sive background knowledge of ATC rules and
procedures are more likely to be selected as
supervisors, while the opposite proved to be true
in the en route environment. However, there are
many other factors that may have entered into the
selection process that were not examined in this
study.

While ¢ of the correlations between the selec-
tion and experimental test scores and the various
SIDP measures for ATCSs in the en route option
were statisticaily significant, they were all quite
low (see Table 2). Of the possible comparisons,
significant correlations were noted between the
MCAT and TMC with the TECH peer-supervisor
zatings (.07 and .06) and between ABSR and SBI
TOTAL (.10). Of the experimental tests, only the
DRT exhibited 2 significant correlation with any
of the SIDP measures; a correlation of .16 with
TECH. Thus, there is weak evidence that some
aspects of cognitive ability at the time of entry
into the ATCS occupation are predictive of
subsequent technical ratings and likelihood of
being selected as a supervisor.

1

As was true for the en route option, correla-
tions between aptitude scores and SIDP measures
for terminal option applicants were low (see
Fable 3). Ratings of communicstion skills were
positively correlated with ABSR (r = .06,
p<.05) and DRT (7 =.22, p<.05) but nega-
tively correlated with HT (r = -.17, p<.05). A
higher rating on interpersonal skills was as-
sociated with a lower score on the OKT & =
.06, p< .04) but higher ratings on ABSR (F =
08, p<.05) and DRT (7 = .18, p<.05). Tech-
nical competence was significantly related to
DRT scores only (7 = .22, p< 05).

Results of these analyses reveal only limiced
evidence of any relationship between any of the
aptitude measures and either the peer-super-
visory ratings or the overall SBI rating. How-
ever, of all the possible correlations, the MCAT
and DRT were most closely correlated with peer-
supervisory ratings 2 of the technical (TECH)
competence of the SIDP applicants in comparison
to the other ratings. This finding was coasistent
across both options. The Abstract Reasoning Test
{ABSR), which measures a related yet different
aspect of the cognitive ability of ATCSs, tended
to be more closely correlated with some of the
other ratings, including the overall SBI rating
for en route ATCSs.

DISCUSSION

These results offer weak support for the con-
cept that measures of aptitudes for the ATCS job
taken prior to entry into ATCS training are
predictive of on-the-job ratings of technical
proficiency. These findings are of increased
interest when one considers the amount of
time, training, and other activities that have
transpired between initial selection and subse-
quent application to become a supervisor. Addi-
tionally, as trainees complete the various stages
from selection through attainment of the full
performance level status, continued iess of
u.successful trainees ensures that the remaining
workforce is relatively homogenous with respect
to technical competence. Perceptions of on-the-
job performance are therefore more likely due to
the presence of untapped aptitudes or differences
associated with various personality dimensions
(e.g., self confidence, assertiveness). In any


Table 9
Correlations between SIDP rating dimensions
and training measures for Terminal Radar Approach controllers

Training Measures

Years to Instructor OT OT
SIDP Ratings FPL Status Rating Hours Days
PSA Ratings (V = 304)
Composite -12° 03 -.157 -.10
Communication -.07 00 -15~ -.06
Interpersonal -.03. -.03 ~1F -.06
Leadership -13" 02 -14 -.10
Technical -2i" Bs) -AT 1g
SBI Ratings (V = 220)
Agency Programs 05. -.00 02 .00
Supervisory Role 13 09 -.06 -.05
Prob Solv/Analytical 2 -.04 Al 07
Judgment 05 -.04 AS 05
Decisiveness 03 -.03 08 -.01
Ocganizing/Planning 06 -.06 13 09
Interpersonal Skill 01 -.07 “12 05
Communication 06 -.10 08 02
Direction/Motivation 01 -.07 05 -.01
pes “ps oi


of time required to reach FPL status and times to
complete specific phases of training for students
who were assigned to comparable facilities. Such
analyses were conducted for controllers assigned
to en route, terminal approach control, and tower
cab facilities.

RESULTS

Results will be discussed in the context of
research questions that address factors which are
relevant for different groupings of controllers.

Question 1: For the combined group of all
controllers, how does proficiency in technical
training relate to success in SIDP?

The only measures of technical training
performance relevant for all controllers are
instructor ratings and overall training success.
Table 1 shows the relationship between instructor
fatings and status in the different stages of the
SIDP process. Those referred to the SBI had
significantly higher instructor ratings than those
not referred, 1(1217)= 2.37, p < .02, and those
selected to be supervisors had significantly higher
instructor ratings than those not selected, (672)
= 2.46, p < .02. However, there was no
difference in the instructor ratings as a function
of SBI eligibility for consideration status.

Table 2 shows the relationship between
training success and SIDP status. The likelihood
of being successful in training was independent
of referral to the SBI, x’(1) = .0005, p > .98,
and was also independent of eligibility for con-
sideration, 7°(1) = 1.98, p > .15. However,
training success was related to supervisory
selection, x"(1) = 5.54, p < .02. A higher
percentage of those who were selected for
supervisory positions were successful in training
at their first facility than those who were not
selected.

Logistic regression analyses were conducted
to assess the relative contribution of both
instructor rating and training success in
predicting status in the 3 stages of the SIDP
program. Using a forward selection procedure,
it was found that adding the mean instructor
rating to a model containing no variables was 2

28

significant improvement in predicting referral for
the Skill Based Interview, x°(1) = 5.07, p <
.03, while adding training status to that model
provided no significant improvement, x’(1) =
.18, p > .65. Neither mean instructor rating nor
training status contributed significantly to the
prediction of eligibility for consideration. Adding
training status to a model containing no variables
significantly impzoved the prediction of
supervisory selection, 7(1) = 9.02, p < .03,
but adding mean instructor rating to that model
did not add significantly to the prediction, x“(1)
= 2.44, p > .10.

It must be mentioned that the goodness of fit
of the model is somewhat questionable; the -2
Log Likelihood (-2 LL) statistics, indicating how
well the model fits the data, were significantly
different from 1 for the prediction of SBI
Referral and supervisory selection; -2 LL (SBI
referral) = 1344.55, p < .01, df = 1218, and
-2 LL(supervisory selection) = 752.49, p < .02,
df = 673. However, the goodness of fit statistics
for the same models did not lead to the rejection
of the models; Goodness of fit (SBI referral) =
1219.0, p > .48, df = 1218, and Goodness of
fit (supervisory selection) = 674.0, p > .48, df
= 673. The prediction of eligibility for con-
sideration by the logistic regression model
resulted in a classification of all applicants as
"eligible," and the prediction of selection by the
model resulted in a classification of ali candidates
as “not selected." Thus, while the model
successfully predicted EFC and supervisory
selection in a statistical sense, the resulting
classifications predicted by the model are of
questionable utility.

Question 2: For en route controllers, de any
of the specific measures of training perfor-
mance predict success in SIDP?

The measures of technical training perfor-
mance relevant to the analyses concerning the en
route option included time to reach FPL status
(in vears), the number of calendar days and OJT
hours required to complete training, and mean
instructor ratings in OJT. Table 3 shows means
and standard deviations for the field training
performance measures by status in each stage of
the SIDP program. Those referred to the SBI had



Table 14
Results of regression analyses using
field training performance measures as predictors of
component rating dimension scores in SIDP Tower Cab controllers

Criterion Predictors R R B F
SBI Ratings
Agency Programs ns
Supervisery Koie ns
Probiem Solving Rs
Judgment ns
Decisiveness ns
Organizing i Years to FPL 73) 04 -.20 6.12
Interper Ski ns
Communication ns
Direction/Motiv Years to FPL 17 03 +17 4.04

45


failure, but students occasionally withdraw from
the training program of their own accord.

Despite the measurement problems associated
with these training performance measures, they
are the best measures currently available to
describe performance in ATCS technical training
programs. Some of the measures described
above have been used as criiciia against which to
validate ATCS selection procedures (Manning,
Kegg, & Bryant, 1989). Statistically significant
(though somewhat small) correlations were found
between the Office of Personnel Management
rating and training status, instructor ratings, and
time to reach FPL (a negative correlation) for
students in the en route option; somewhat higher
correlations were found between the Academy
score and the same measures of field training
performance.

For students assigned to terminal radar
approach control facilities, the OPM score pre-
dicted training status, instructor ratings, and time
to reach FPL, while Academy scores only pre-
dicted training status and instructor ratings. For
Students assigned to tower cabs, the OPM score
was not predictive of any measure of field train-
ing performance. However, the Occupational
Knowledge Test score, which adds extra credit
points to the earned rating for those who demon-
strate job knowledge, was significantly correlated
with days and hours in training phases, as well
as time to reach FPL status and instructor rat-
ings. The Academy score was significantly
correlated with all training times and instructor
ratings for students at tower cab facilities.

For this study, one question of interest was
how well the available measures of field training
performance would predict the ratings which
determined whether the candidate progressed to
the next stage of supervisory selection. It was
expected that measures reflecting technical
proficiency, such as training status or times to
complete certain portions of the training
program, would predict ratings of technical
competence used in the Peer-Supervisory
Assessment (PSA) phase of the SIDP but would
be unrelated to other types of ratings. On the
other hand, instructor ratings of student potential
might be related to ratings reflecting other

27

qualities of the candidate, such as interpersonal
skills.

Another question of interest was how well
the available measures of field training perfor-
mance would predict success in the 3 stages of
the supervisory selection process. It was also
expected that instructor ratings of student
potential might be predictive of referral to the
Skill-Based Interview (SBI), which is based on
the results of the PSA. On the other hand, it
might be expected that none of the measures
would igibility for Consideration (EFC), an
assessment made by an independent group of
Taters.

METHOD
Subjects

Records of field training performance were
obtained for 1,352 applicants to the SIDP pro-
gram who entered the FAA Academy between
August 1981 and September 1985, successfully
completed that program, and entered ATCS field
training. Three records were discarded because
the field training data were unavailable. For

other records, field training data were
occasionally incomplete, resulting in some
missing data.

Training Performance Measures

Because of the differences in the amount and
content of training provided to ATCSs assigned
to different types of facilities, some measures of
field training performance should not be
compared for the 3 types of air traffic
controllers. The only available measures of
training performance appropriate for assessing
general technical competence are instructor
ratings and whether or not the controller was
successful in training at his or her first facility.
(Controllers who are unsuccessful in their initial
assignment may be reassigned to a facility with
lower air traffic count and complexity. Some
controllers who failed at their first facility could
eventually reach FPL status at a subsequent
facility and eventually apply for the SIDP
program.) However, it is reasonable to analyze
the relationships between more specific measures
of field training performance, such as the amount


APPENDIX A
FIRST-LINE SUPERVISOR COMPETENCIES

ORGANIZATIONAL KNOWLEDGE: Demon-
sates knowledges of the FAA organizaticnal
components, the mission(s) of each relevant
organizational unit, and the principal programs in
the FAA.

KNOWLEDGE OF SUPERVISORY ROLE
PERFORMANCE: Displays knowledge of the
roles, responsibilities, and duties of supervisors
and managers; accurately assesses the impact
upon others of role performance; and supports
and promotes organizational decisions, policies,
programs and initiatives such as EEO, Employee
Assistance Program, Survey-Feedback-Action
Program, and Affirmative Action.

COMMUNICATION SKILL: Presents end
expresses ideas and information effectively and
concisely in an oral and/or written mode; listens
and comprehends what others are saying; shares
inforiaation with others and facilitates the open
exchange of ideas and information; is open,
honest, and straightforward with others; provides
a complete and timely explanation of issues and
decisions in a manner appropriate for the aud-
ience; and present information and material in 2
manner which gains the agreement of others.

DECISIVENESS: Makes decisions, renders
judgments, and takes action on difficult or un-
pleasant tasks in a timely fashion, to include the
appropriate communication of both negat.ve and
positive information and decisions.

DIRECTION AND MOTIVATION: Motivates
and provides direction in the activities of others
to accomplish goals; gains the respect and con-
fidence of others; appropriately assigns work and
authority to others in the accomplishment of
goals; provides advice and assistance as required;
and establishes high quality work standards for
self and others.

TECHNICAL COMPETENCE: Understands
and appropriately applies procedures, require-
ments, regulations, and policies; maintains
credibility with others on technical matters; and
uses equipment, procedures, or systems in the

operational and/or staff environment as the
position requires.

INTERPERSONAL SKILL: Is aware of, re-
sponds to, and considers the needs, feelings, and
capabilities of others; deals effectively with
others in both favorable and unfavorable situa-
tions regardless of their status or position; ac-
cepts interpersonal and cultural differences;
Manages cenflicts, confrontations, and disagree-
ments in a positive manner which -ninimizes
personal impact, to include controlling one’s own
feelings and reactions; and provides appropriate
support to others.

JUDGMENT: Develops and evaluates alternative
courses of action; makes decisions based on
correct assumptions concerning resources and
guidelines; supports decisions or recommenda-
tions with data or reasoning; defines and imple-
ments solutions to problems; and recognizes
when no action is required.

PLANNING AND ORGANIZING: Identifies
requirements, allocates, and effectively uses
information, personnel, time, and other resources
necessary for mission accomplishment; es-
tablishes appropriate courses of action for self
and/or others to accomplish specific goals;
develops evaluation criteria and tracking systems
for monitoring goal progress and accomplish-
ment; and specifies objectives, schedules, and
priorities.

PROBLEM SOLVING AND ANALYTICAL
ABILITY: Identifies existing and potential
problems; notes, understands and includes the
critical elements of problem situations; obtains
and evaluates relevant information; demonstrates
awareness that new and/or additionai information
sources are required; notes interrelationships
among elements; identifies possible causes of the
problems; recognizes the need to shift to an
alternative course of action including innovative
or creative approaches; and appropriately ter-
minates information collection and evaiuation
activities.



sample, or three-quarters of those referred to the
SBI, were rated as "Eligible for Further Con-
sideration” in the SIDP. Just 14% (N = 160) of
the total sample, or 26% of the persons rated as
“Eligible for Further Consideration,” were
selected as first-line supervisors as of March
1991. Table 1 presents data on applicant status in
SIDP by career option.

Measures

Thirteen performance measures were collec-
ted on each student during approximately 9
weeks at the FAA Academy. Five of the meas-
ures were multiple-choice tests covering the
academic materiais. These assessed the candi-
date’s ability to learn and retain the basic know-
ledge required for the job. A sixth measure was
a map test, which assessed the student’s ability to
learn a map of the relatively simple synthetic
airspace. These 6 measures accounted for 10% of
the student’s final score. For purposes of this
paper, scores from 4 multiple-choice tests and the
map test were combined into a Block Test Aver-
age (BA). The fifth multiple choice test was a
comprehensive final exam on the academic
phase, the Comprehensive Phase Test (CPT), and
was analyzed as a separate measure.

In the laboratory phase, six 30-minute stan-
dardized laboratory problems were formally
graded. A student received two scores from a
grading instructor on each problem. The first
score, the technical assessment (TA), was a
numeric assessment of the student’s errors in
application of ATC rules and procedures. The
second score, the instructor assessment (1A), was
a subjective instructor rating of the student’s
globa: performance on the problem. These 2
scores were averaged for the final problem score.
The student’s lowest score of the 6 laboratory
problems was dropped. Thus, only 5 of the 6
graded problems were counted toward the final
comprehensive course score. The graded labor-
atory simulation problems comprised 65% of the
final score. Finally, the students were given a
timed, multiple-choice test, the Controller Skills
Test (CST), which was a paper-and-pencil assess-
ment of the student’s ability to apply ATC rules
and procedures. This test comprised 25% of the
Final Comprehensive Score (COMP). Table 2

18

summarizes these FAA Academy initial qualifi-
cations training performance measures.

Analyses

Because the En Route and Terminai Initial
Qualification Courses were different programs
utilizing measures similar in name only, analyses
were conducted for the En Route and terminal
options separately. Analyses were conducted to
address the following research questions:

Research Question 1

What was the relationship between FAA
Academy technical performance and PSA ratings,
especially the technical competence PSA rating?
We hypothesized that there would be a non-zero
correlation between the FAA Academy measure
of technical performance and PSA ratings, par-
ticularly the PSA technical competence category
or dimension.

Research Question 2

Was performance in FAA Academy initial
technical qualification training better for suc-
cessful than for unsuccessful SIDP applicants at
each stage of SIDP? We hypothesized that there
would be significant differences in FAA Acad-
emy scores for successful compared to unsuc-
cessful participants at each stage of SIDP. We
used multiple analysis of variance (MANOVA) to
test this hypothesis.

Research Question 3

Did performance in the FAA Academy ini-
tial ATCS qualification training program predict
success at each stage of SIDP? We hypothesized
that higher performance in the FAA Academy
would improve the odds for success ;< each stage
of SIDP. We used logistic regression (Hosmer &
Lemeshow, 1989; Norusis, 1990) to test this
hypothesis.

RESULTS

Research question 1: Academy performance -
PSA ratings


Tabie 4
Descriptive statistics for predictors and
SIDP rating dimensions for En Route controllers

Measures M SD
Training Performance
Instructor Rating 4.25 53
Years to reach 2.60 59
O}T Hours 235.33 88.68
OIT Days 142.17 31.67
PSA Ratings
Composite 61.04 15.52
Communication 62.01 15.34
Interpersonal 60.34 17.57
Leadership 60.22 16.91
Technical 61.59 15.76
SBI Ratings

Agency Programs 3.66 83 313
Supervisory Role 3.84 381 313
Problem Solv/Analytical 3.96 89 313
Judgment 3.82 94 313
Decisiveness 4.07 87 3B
Organizing/Pianning 3.93 31 31
Interpersonal Skill 4.21 85 313
Communication 4.06 89 313
Direction/Motivation 3.94 93 313



Table 5
Correlations berween SIDP rating dimensions
and training measures for En Route controllers

Training Measures

Years to Irstructor O}T OT

SIDP Ratings FPL Status Rating Hours Days

PSA Ratings (N=435)
Composite -.10" 247 -.207 -.18"
Communication -.08 27 -.187 157
Ynterpersonal ~.06 197 -.167 -.16~
Leadership iy 2 -.207 “AT.
Technical ~.147 3r -.20" -.22

SBI Ratings (N=291)
Agency Programs 01 02 -.08 00
Supervisory Role “13° 01 -.01 -.02
Prob Solv/Analytical 07 -.05 00 -00
Judgment .08 -.05 -.07 -.09
Decisiveness 02 04 -.06 -AL
Organizing/Planning 09 -.06 -.04 -.05
Interpersonal Skill -.01 -.05 02 -.08
Communication 03 -.01 -.03 .08
Direction/Motivation OS -.09 00 06

ps Ol p< 001

36



(yh = N) 10> Tae SOS dy
si +1 el a U 01 6 8 L 9 s * € z 1 sounsvoj,

P<: ae > 4 A 2 A YS 9 £0" 80" 80" SO ANOW/IC (SI
7 =09) SS OS'S’ SES oe 8 0- = $0" 80° 90° sO 00'- NON ED]UNUWOZ (p]
SS a SS, i: || co |) | <a jeuossodioyuy (1

~69 = 89 TLE SES 10 SO OV 2 00 0" BQ /Buyuuryd (ZI

89 = 69" SE 6" $0" ay to 0° 10° oayspood (TT

wl ve TE’ £0 90" so" ny y- $0" quowSpny (01
67 670-0" 90° 9 8610'S 0 BujAjos qoid (6 3

5S (0 60" 6 I 60> £0" ajoy adng (g

sot 4 a) a ( y= adpojmouy 810 (L

rn  : nS e 5 2 Teouypay (9

wll’ 28" S097 dyyssapeoy (¢

wth 60°. Teuosiadsoiny (py

90'- .1Z — UopeajUNWUIOD (¢
00" Td SHOW (Z
Buney $49 (1

s8uyivs [gg pun ‘s8upsos yo ‘sainsvau souvuuofiad joouysat Suown suopvjassoD
1 2148,


TABLE 4
Comparison of FAA Academy performance measures by ATC option and outcome at each stage of SIDP

Selection

ible for Further
‘onsideration

e

Eli
Not Eligible

Referred to SBI

Not Selected Selected

Eligible

Referred

Not Referred

SD

sD

Measure

Option

THAOMON!

Soar aaa
adam
Arona

WMD OMO
seat

sane
BRES

wmMnammne

AHaAAANHS

gee Base

BA
CPT
AVLS
AVTA
AVIA
COMP

En Route

24

mune neanw

96.6
90.2
78.0
64.9
79.8
81.0
80.1

an onmawn

gageses

BESES5

anonnan
mMseste

Besse

MN OANA

MANNA
ygersss

anernas

oanenas
esers-s

ASWROBDNS
AArOMAN

MOWMACE
rag
S2se

< oe

555558

SezeUg
3
&
:
&

¥p < 05, **p < 01



Table of

Contents

RS

Au Overview of the Air Traffic Control Specialist
and First-Line Supervisor Selection Systems
Jennifer G. Myers, Ph.D... 2.0.0.0... 000 ce eee eeee 1

Cognitive Indicators of ATCS Technical Ability and
erformance in a Supervisory Selec:ion Pregram
David J. Schroeder, Ph.D.

Air Traffic Control Specialist Technical Competence

in Initial Training and Selection as a First-Line

Supervisor

Pamela S. Della Rocco, M.A., and

Dana Broach, Ph.D... 2.0.2... 0... 00. cee eee ee ees 45

Relationships Between Performance in Air Traffic Control
Specialist Technical Training and Supervisory Selection
Programs

Carol A. Manning, Ph.D... 2.02.22. 2. cee eee eee ee 25

Candidate Performance in a Supervisory Selection Program
and Subsequent Selection Decisions
Jennifer G. Myers, Ph.D... 2.2... ee 47

iii



Table 12
Descriptive statistics for predictors and
SIDP rating dimensions for Tower Cab controllers

Measures M SD N
Training Performance
Instructor Rating 4.23 0 232
Years to reach FPL 66 35 240
OJT Hours 104.17 44.81 238
OST Days 162.65 103.38 247
PSA Ratings
Composite 57.12 15.52 251
Communication 59.10 15.88 251
Interpersonal 56.84 17.52 251
Leadership 55.65 17.18 251
Technical 56.83 16.62 251
SBi Ratings

Agi Programs 3.49 88 162
Supervisory Role 3.75 86 162
Problem Solv/Analytical 3.85 1.02 162
Judgment 3.66 93 162
Ftoomcheney 353 tot 1e2
izi janning . HT 162
Interpersonal Skill 4.05 87 162
Communication 3.88 99 162
Direction/Motivation 3.76 98 162

43


Results of regression analyses using field trainings

Table 10

performance measures as predictors of 4
dimension scores in SIDP Terminal Radar Approach controllers

x

Predictors R 8 F Pp
PSA Ratings
Composite OJT Hours 24 06 3.24 12.66 001
Communication OJT Hours 21 05 +21 9.95 002
OST Hours -23 05 -.23 11.38 -001
Leadership OJ¥ Hours 19 04 -19 8.14 .005
Technical Inst Rating 28 08 18 8.68 .001
OJT Days +17
SBI Ratings
Sipervisory Role ns
supervisory Role ns
Problem Solving ns
Judgment OST Hours 16 02 16 5.30 03
Decisiveness ns
Organizing OST Hours 14 02 14 3.97 05
Interper Skill ns
‘Communication ns
Direction/Motiv ns

41



T Gano

3S3HL JO INON “CG

OE ONY 02 ‘9
02 GNV Ov ‘8
02 GNV 09 ‘Vv
31V9S 3951
mH ;

21890 210I1INOO THM LIVYOUIY HOIHM
NOILSIND JIdWYS

alyits 08 0008 02

arnia 08~ 0008 09

aduNIG Ove 0059 0S

dt XHO Ove 0069 Ov

arov ove 0004 Of

O02
v3 | a . ; 37998 08p 0004 0z
OHWDY O8p 0004 ol

aLNOw G33dS 3aNLiW = Livuduiv
(LVOIN) LS3L 30N.LILd¥ YA TIOULNOS X32 1d LIN

at



Table 3A
Mean scores by referral status

Referred to SBI Not Referred
W = 746) (WN = 239)
Measures M SD M SD
Performance Rating 4.00 67 3.52 57
Months FPL Experience 44.46 13.75 43.56 14.79
Table 3B
Mean scores by promotability status
Eligible for Consideration Not Eligible
(WW = 549) (W = 197)
Measures M SD M SD
PSA Composite 64.13 10.80 63.07 11.59
Performance Rating 4.01 67 3.98 67
Months FPL, Experience 44.46 13.75 44.52 14.04
Table 3C
Mean scores by selection status
Selected Not Selected
(N = 13%) (N = 410)
Measures M SD M SD
PSA Composite 67.03 11.38 63.16 10.44
SBI Knowledge Composite 8.09 1.45 791 67
SBI Skill Composite 30.61 2.90 30.33 2.84
Performance Rating 4.12 .70 3.97 66
Months FPL Experience 45.20 13.03 44.18 13.83

*U.S.GP0:1992-661-063/40076


€ aanoL

ozst st o'Sst g's} GLSt SgNnLiliv ‘€
O¢- o'oL- OVE gst- o'oL SAYuadWNV “2%
o'sol O'%6 36 368 Or6 ‘Wid ‘ut

a a ce] q Vv *SW3180u"d J1dNVS

olva
uv-Tand

os

ot
(sanvsn y
matin

oe 0%

Isai SNIGVAd Id

13



Techriccl Report Documentation Page

1. Report No. 2.” Government Accession No. 3. Reciprent's Catalog No.

DOT/FAA/AM-92/16

4. Title and Subtitle

5. Report Dare

April 1992

A LONGITUDINAL EXAMINATION OF APPLICANTS TO THE P

AIR TRAFFIC SUPERVISORY IDENTIFICATION AND 6. Performing Organization Code |
DEVELOPMENT PROGRAM.

T8.” Performing Organtzation Report No.

7. Author's}

Jennifer G. Myers, Ph.D., Editor

9%. Performing Organization Nome and Address 10. Wark Unit No. (TRAIS)

FAA Civil Aeromedical Institute

P. O. Box 25082 TT, Contract or Grant No.

Oklahoma City, OK 73125

. Type of Report and Period Covered

12. Sponsoring Agency Nome and Address

Office of Aviation Medicine

Federal Aviation Administration
Independence Avenue, S.W. 14. Sponsoring Agency Code

Washington, D.C. 20591 Pensering Agency ©

15. Supplementary Notes

This work was performed under task AM-C-92-HRR-125.

16, Abstroct

The Federal Aviation Administration began development of an extensive longitudinal database on its air traffic
controller workforce following the strike of 1981. Since that time, data have been collected on thousands of air
traffic controllers, spanning a period which covers their application to the federal government for employment
to their achievement of a first-line supervisor position. This collection of papers examines a subset of air traffic
control specialists who have completed the agency’s supervisor selection program, beginning with their
performance on the Office of Personnel Management test battery and other cognitive tests administered prior to
completion of the air traffic controller Screen Program. Measures of academic, laboratory, and overall Screen
performance were examined in relationship to aspects of performance in the supervisor selection program.

ield training profiles were analyzed to determine differences between successful and unsuccessful supervisor
selection program candidates and relationships with selection program performance. Finally, performance in the

supervisor selection program was compared for those who were selected as first-line supervisors and those who
were not.

17. Key Words 18. Distribution Statement
Selection, Air Traffic Control Document is available to the
Specialist, First-Line public through the National
Supervisor, Performance, Technical Information Service,
Training Springfield, Virginia 22161
19. Security Classif. (of this report) 20. Security Cla_ssf. (of this poge) | 21. No. of Pages 22. Price
L Unclassified Unclassified ST

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized


OIL O'ZIZ 668 PSEC 998 COEZ 86 OTEZ CRB COEZ $88 8'0Sz SINOH LO
v6y «OZIT O'R «= L'ZHLZLL ORT «RG BOTOCCER:«OS'OET, «= E'9L_—S'0ST skeq L{0
s v2 9 9% oF OF OF st 9 ©8699 9 Le Idi 01 S109,
9 vy OS ve OS te 69 ye 6S Oey s ea Bupey soronsuy
as ow as ww a Ww a Ww a oew aeowW sornseayy SuureryL, Plata
parooyas pawo]9g 10N aiarsia 0191811 10N poxojoy pe130}04 30N
uonoejeg Asosjasedng woNEJep|suOD JoyN,] IGS 03 posgyoy

Joy o1q131g

S49]JOLUOD aINOY UT AQIS Uf Snivis puv aouvuuiofied 8ujujvas pjayf ueanseg diysuopvjay

€ 1981

34



Patton, J. A. (1974). The foreman: Most misused
person in industry. Management Review,
63{11), 40 - 42.

Phillips, J. J. (1985). Jmproving the supervisor’s
effectiveness. San Francisco: Jossey-Bass.

Raza, S. M. (1987). Personality characteristics
of effective first line supervisors. Unpub-
lished dissertation, the University of Tulsa.

Seige!, A. I. (1978). Miniature job training and
evatuation as a selection/classification device.
Human Factors, 20, 189 - 200.

Seigel, A. I. (1983). The miniature job training
and evaluation approach: Additionai find-
ings. Personnel Psychology, 36, 41 - 56.

Seigel, A. I., & Bergman, B. A. (1975). A job
learning approach to performance prediction.
Personnel Psychoicgy, 28, 325 -339.

TABLE 1
Sample status at each stage of SIDP by ATC option

Option
Status En Route Terminal
Referral to Skill-Based Interview (SBD
Referred 351 494
(75%) (75%)
Not Referred 117 166
(25%) (25%)
Eligible for Further Consideration
Eligible 252 363
(72%) (73%)
Not Eligible 99 131
(28%) CT1%)
Selected for a Supervisory Position
Selected 34 106
(21%) (29%)
Not Selected 198 257
(79%) (71%)

22


NOTICE

This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents or use thereof.


CANDIDATE PERFORMANCE IN A SUPERVISORY SELECTION PROGRAM
AND SUBSEQUENT SELECTION DECISIONS

Jennifer G. Myers, Ph.D.

Smail surveys of private organizations have
reported that fewer than half of these organiza-
tions have forma! selection systems in place for
firstline supervisors (Levine, 1986; Rendero,
1980). Given the importance of first-line super-
visors in énsuring employee productivity by
directing subordinate work activity, as well as
their role in representing management and the
organizational mission to subordinates, the lack
of reported formal supervisory selection systems
is somewhat surprising. Recent research on
supervisory selection systems and promotion
decisions is also noticeably absent.

A recent review of federal agencies’ super-
visor selection programs (U. S. Merit Systems
Protection Board, 1989) described the typical
situation. In general, applicants to first-line
supervisory jobs document training and exper-
ience information typically based on their prior
nonsupervisory work history. As noted in the
teview, the problem for the selecting official
becomes one of evaluating an individual’s ability
or potential for performing supervisory tasks that
have never been done before. Given the makeup
of the application materiais, selecting Officials
primarily see the applicant’s prior technical work
experience. Personal interviews with applicants
may be restricted by the associated costs, since
Merit Promotion Pian regulations require all or
none of the eligible candidates to be interviewed.
The frequent result in the federal government is
that the most highly technicaliy qualified person
is chosen for the supervisory job, rather than
someone with demonstrated management skills.
This outcome is similar to that found in private
industry (Northrup, Cowin, Vanden Pias, Ful-
mer, Bolick, Bellace, & Rosenzwig, 1978).
However, selections based on technical perfor-
mance may result in the promotion of someone
unsuitable for the position, since technical skills
and experience are not necessarily related to
supervisory competence (Phiilips, 1985).

47

Technical competence has been identified as
an important factor in the performance of
supervisory tasks (Myers & Stutzman, 1991).
However, Myers and Stutzman (1991) found that
other competencies such as communications skiils
and interpersonal skills were mentioned more
frequently than was technical competence as
important in successful task performance. One
method of ensuring the applicant for a first-line
supervisory position has the right mix of
technical and supervisory skills is to formalize
the assessment of the mest important
competencies through a structured selection
program. Still, selecting officials could
emphasize the importance of technical over
supervisory competence, by selectively attending
to factors in the work history that demonstrate
the applicant’s technical competence. As already
suggested by Phillips (1985), this may not be an
optimal strategy for selecting the best candidate
for the position. As one example, a study of
engineers (Slusher, Van Dyke, & Rose, 1972)
found that the most highly technically qualified
supervisors were least likely to adopt a
managerial role and in turn, were lowest in
group productivity among the groups studied.

As noted in the introduction of this technical
report, the Federal Aviation Administration
implemented the Air Traffic Supervisory
Identification and Development Program (SIDP)
to change the emphasis on technical performance
in promotability decisions to inchide other skills,
such as decision-making and communication, that
reflect supervisory potential. In order to under-
stand how the different competencies may
distinguish between successful and unsuccessful
candidates at different phases of the SIDP,
comparisons were made of candidate performance
on SIDP performance measures, as well as
measures of technical performance and
experience. The influence of technical
performance and skills measured in the SIDP on
success in each of the phases was also assessed.



terminal radar approach controllers included time
to reach FPL status (in years), calendar days and
OJT hours required to complete radar training,
and an instructor rating in radar training. The
training days and OJT hours were adjusted by
dividing them by the number of radar positions
in the airspace for which their facility was re-
sponsible.

Table 7 shows means and standard deviations
for the field training performance measures by
status in each stage of the SIDP program. Those
referred to the SBI took significantly fewer years
to reach FPL status than did those who were not
referred, F(1,314) = 4.33, p < .04. Logistic
Tegression analyses were then conducted to assess
the relative contribution of the training measures
in predicting status in each phase of SIDP.
A model was developed to predict SIDP status,
using the measures described above as indepen-
dent variables and the 3 stages of advancement
in SIDP as the dependent variables. A model
containing time to reach FPL status marginally
predicted referral to the Skill-Based Interview,
x(1) = 3.74, p < .06, while instructor rating,
OST hours, and number of days spent in radar

ing did not contribute significantly to the
model, x3) = 4.09, p > .25. The training
variables did not contribute significantly to the
prediction of eligibility for consideration, nor did
they contribute to the prediction of supervisory
selection.

Although both goodness of fit statistics for
the model predicting referral for the SBI
suggested that the model fit the data -2 LL =
303.57, p > .26, df = 289), the prediction of
eligibility by the model resulted in a classification
of ali candidates as "eligible." Thus, while the
model successfully predicted EFC status in a
statistical sense, the classifications are not useful.

Question 5: For terminal radar approach
controllers, in what ways are the measures of
training performance related to the rating
dimensions used to determine success in SIDP?

Table 8 shows descriptive statistics for all
predictors and criteria. Table 9 shows correla-
tions between the predictors and SIDP rating
dimensions. The number of years to attain FPL

30

status was significantly correlated with several
of the PSA rating dimensions, but only with the
Supervisory Role rating dimension of the SBI Gn
the opposite direction than was predicted). OJT
hours also had significant negative correlations
with the PSA rating dimensions but were posi-
tively correlated with the Judgment rating dimen-
sion of the SBI.

Regression analyses, shown in Table 10, were
conducted to assess the relative contribution of
the training measures in predicting each of the
rating dimensions in the SIDP program. Mea-
sures of technical training performance were
agai’ more highly related to ratings made by
peers and supervisors than ratings made by those
conducting the Skill-Based Interview. For radar
approach controller cardidates, OJT hours sig-
nificantly predicted ratings on all the PSA scales
except for the Technical scale, which was predic-
ted by a model containing both instructor ratings
and time to compiete training. Regression models
containing OJT hours also predicted ratings on
the SBI Judgment and Decisiveness scales, but
the relationship was not in the predicted direc-
tion. None of the models containing measures of
training performance accounted for more than
10% of the variance in the ratings.

Question 6: For tower cab controllers, do any
of the specific measures of training
performance predict success in SIDP?

The measures of technical training perfor-
mance relevant to the analyses dealing with tower
cab controllers included time to reach FPL status
(in years), calendar days and OJT hours required
to complete training on the local control position,
and the mean instructor rating for OJT. Those
referred to the SBI had significantly higher
instructor ratings than did those not referred,
F(i,230) = 5.14, p < .03. No significant
differences in field training performance were
observed as a function of eligibility for
consideration or supervisory selection. Logistic
regression analyses were conducted to assess the
relative contribution of the training measures in
predicting status in each phase of SIDP. A model
was developed to predict SIDP status, using the
measures described above as independent
variables and status in the 3 stages of SIDP as


Table 8
Descriptive statistics for predictors and SIDP
rating dimensions for Terminal Radar Approach controllers

Measures M SD N
Training Performance
Instructor Rating 4.04 84 363
Years to reach FPL 1.84 87 316
OJT Hours 62.04 40.60 338
OJT Days 95.06 66.65 339
PSA Ratings
Composite 57.66 13.77 378
Communication 59.49 13.46 377
Interpersonal 57.58 15.32 378
Leadership : 56.38 15.74 378
Technical 57.38 15.42 378
SBI Ratings

Ag Programs 3.84 719 270
Supervisory Role 3.88 76 270
Problem Solv/Analytical 4.00 39 270
Judgment 3.84 89 270
Decisiveness $a 34 270
Organizing/Planning . 70
Interpersonal Skill 4.24 83 270
Communication 4.33 80 270
Direction/Motivation 3.96 91 270

39



RELATIONSHIPS BETWEEN PERFORMANCE IN AIR TRAFFIC CONTROL SPECIALIST
TECHNICAL TRAINING AND SUPERVISORY SELECTION PROGRAMS

Carol A. Manning, Ph.D.

After successfully completing the Federal
Aviation Administration’s (FAA) 2-stage selec-
tion procedure, as described in Della Rocco and
Broach (1991), Air Traffic Control Specialists
(ATCSs) enter technical training programs in
their respective field facility assignments. The
type and duration of the training depends on the
type of facility (option) and amount of aircraft
traffic controlled by the facility. A brief
discussion of the types of air traffic control
facilities will be presented, followed by a
description of the training programs provided by
each facility type.

«TCS Options

Air Traffic Controllers in the FAA can be
split into 3 cptions or specialties: en route,
terminal, and flight service station (FSS). En
toute and terminal ATCSs ensure the separation
of aircraft traveling between airports (en route)
and approaching or departing from airports
(terminal) by formulating and issuing clearances
(sets of instructions for aircraft regarding their
appropriate altitudes and directions of flight).
The clearances are designed to ensure aircraft
separation and maximize fuel efficiency. Flight
Service Station specialists provide services to
pilots such as giving weather briefings, filing
flight plans, and helping to locate lost aircraft.
For this study, the relationship between technical
training performance and performance in the
Supervisory Identification and Development
Program (SIDP) was limited to en route and
terminal controllers, because the screening,
training and technical requirements of the FSS
specialist’s job are considerably different than
those for ATCSs in the other options.

ATCS Field Technical Training

In spite of some similarities in job functions,
the technical training provided for en route and
terminal controllers is very different, and even
within different types of terminal facilities, the
training differs. Much of the reason for the dif-

25

ferences in training is that air traffic control
procedures differ according to the facility to
which the student is assigned. At en route
facilities, traffic is usually moving rapidly and at
high altitudes and must be kept further apart than
at terminal facilities, where traffic is slowing
down and converging on a single location. Thus,
separation standards (the minimum distance that
aircraft must be kept apart) vary by type of ATC
facilities and the training varies as well.

In the en route option, the unit of air traffic
control operation is the sector, a piece of air-
space for which a team of 2 or 3 controllers is
responsible (during times of slow traffic, only 1
controller may be responsible for a sector). A
group of between 5 and 8 sectors is combined
into what is called an area of specialization. An
en route controller is assigned to only 1 area of
specialization, but is responsible for learning to
control traffic in all sectors within that area. The
team of en route controllers working at most
sectors handles duties related to radar separation
of aircraft (radar duties, including formulating
clearances to ensure separation and delivering
them by radio to pilots, handing off responsibility
for at: aircraft to another controller), duties to
assist the radar controller (radar associate duties,
including maintaining records about clearances
that have been issued or other changes in the
flight plan of an aircraft, identifying potential
problems, communicating information not
directly related to aircraft separation of aircraft to
pilots or other controllers), or other support
activities (assistant controller duties, including
entering data into the computer, ensuring that all
records of flight progress are available for the
controller in charge). En route controllers are
trained on assistant controller duties first, then
are given training on increasingly difficult re-
sponsibilities (radar associate duties, then radar
duties). Training on concepts is conducted first
in the classroom, then applied in a laboratory
setting, then reinforced during on-the-job training
(OST) conducted in a supervised setting. At some
facilities, all radar associate training is completed



TABLE 1

Descriptive statistics for the OPM selection battery

MCAT. ABSR OKT.
M SD M SD M SD N Group

Applicants
69.7 16.1 30.5 9.5 28.8 11.3 127,807 1981-1985'

En Route
90.0 Ti 39.8 5.9 36.2 12.9 5,993 Academy entrants’
91.6 6.5 405 5.8 37.6 13.2 3,337 Passed Academy
914 65 40.5 5.7 37.6 12.8 1,215 SIDP applicants
90.8 6.5 40.7 5.5 37.5 12.8 468 All SIDP eligibles
90.8 6.6 406 56 38.1 12.9 407 SIDP eligible - not yet selected
91.1 6.1 417 49 33.3 114 61 Selected

Terminal Option

885 = 7.3 39.1 6.3 39.4 13.4 3,095 Academy entrants!
89.4 6.9 39.6 6.1 41.2 13.7 2,186 Passed Academy
895 69 39.4 6.3 40.7 13.4 1,084 SIDP applicants
89.8 6.7 39.4 6.2 40.1 13.1 461 SIDP eligibles
90.2 6.5 39.6 6.1 39.3 12.9 85 SIDP eligible - not yet selected
88.2 7.3 38.7 66 43.6 13.8 85 Selected

NOTES: ‘From Manning, Delia Rocco, and Bryant (1989)


METHOD
Subjects

Subjects were 985 nonsupervisory air traffic
control specialists who entered the Ex Route or
Terminal Screen Program in the Academy be-
tween 1981 and 1985, applied to SIDP between
1985 and 1988, and completed their field training
in an En Route or Terminal facility. The average
age of this group at the time they applied to
SIDP was 32.14 years and the average full
performance level (FPL) experience was 3.69
years. Fifteen percent were female and 85% were
male. All applicants were required to have at
least a fully successful rating on their last
performance appraisal; 29.6% were rated fully
successful, 52.5% were rated exceptional, and
17.9% were rated outstanding.

Mecsures

Peer-Supervisory Assessment. Applicants to
SIDP identify a combination of peers and first-
line supervisors to complete the Peer-Super-
visory Assessment (PSA). A paired comparison
approach is used to rate applicants against other
applicants and benchmarks on 4 different
dimensions: Technical Competence,
Communication Skills, Interpersonal Skills, and
Leadership. Scores are computed on a 0 to 100
scale; the composite PSA score is a simple
arithmetic mean of the 4 dimension scores.
Successful completion of this hurdle results in
referral to the next step in the SIDP process, the
Skiil-Based Interview (SBD.

Skili-Based Interview. The SBI is a combin-
ation of a face-to-face interview and 3 role-
plays. Three interviewers provide consensus
ratings on a 1 (weak) to 5 (outstanding) scale on
the following 9 dimensions: Organizational
Knowledge, Knowledge of Supervisory Role
Performance, Organizational Planning, Problem-
Solving and Analytical Ability, Judgment,
Decision-Making, Communication Skills, Inter-
personal Skills, and Leadership. Based upon the
combination of ratings received during the SBI,
individuals are determined to be (1) “promotable”
and placed on the Eligible for Consideration
(EFC) list, allowing them to apply for first-line
supervisor positions, (2) in need of further

skill remediation prio: to placement on the EFC,
or (3) recommended for further self-development
and reapplication to SIDP at a later date.

Journeyman Level Technical Performance.
Technical performance at the journeyman level
was identified through 2 broad measures: (a) the
number of months of journeyman level ex-
perience the applicant had at the time of applica-
tion to SIDP and () the most recent performance
appraisal rating prior to application to SIDP.
Although performance ratings can actually range
from 1 (unacceptable) to 5 (outstanding), the
program requirement for a fully successful (3)
rating to apply to SIDP restricted the range to the
3 highest possible ratings.

Two different sets of analyses were con-
ducted to examine 3 different groups: (a) whether
the individual was referred to the SBI or not
(Referral), (b) whether or not the applicant was
placed on the Eligible for Consideration list
(Promotability), and (c) whether or not the
person was selected for a first-line supervisor
position (Selection). Multivariate analysis of
variance (MANOVA) was used primarily as
“protection” from an inflated alpha level on the
univariate tests comparing PSA and SBI com-
posite ratings and technical performance mea-
sures between the groups. Although statistically
Significant differences are not unexpected, pri-
marily because of the large sample size, these
differences may not have a meaningful impact
on the likelihood of success in the different
phases of SIDP. Thus, iogistic regression
analyses were conducted to determine the relative
influence of the SIDP and performance variables
on the probability of dichotomous (i.e, "pass-
fail") SIDP outcomes.

RESULTS

Correlations were computed between each of
the dimensicns rated in the SIDP. High correla-
tions among PSA ratings and among SBI ratings
(see Table 1) suggested that composite variables
for the PSA and SBI might be needed. Separate
factor analyses (principle components and
orthogonal rotation) were conducted for the PSA
and SBI dimension ratings. Only 1 factor was
found for the PSA; scores on each dimension


mance Rating, and Months FPL Experience. The
MANOVA test was statistically significant (Pi-
liai’s Trace criterion, F=4.00, df=5,543
p<.001). ANOVA tests demonstrated signifi-
cantly different group means on the PSA Com-
posite (F= 13.56, df=1,547, p< .001), the SBI
Knowledge Composite (F=4.67, df=1,547,
p<.05), and Performance Rating (F=4.74,
df=1,547, p< .05).

The logistic regression analysis included all
the variables used in the MANOVA. The results
showed that only the PSA Compusite entered
intothe equation for predicting selection for a
supervisory position. Assessment of the good-
ness-of-fit of the model indicated a poor fit to the
data 62(1,547)=605.78, p=.04). Although the
overall classification rate was 74.86%, the model
predicted that no one had been selected for a
supervisory position.

DISCUSSION

The results of the analysis of the referral
variable identified a significant difference bet-
ween the Referred and Not Referred groups on
the PSA Performance Ratings. In addition, the
PSA composite and performance rating predicted
teferral to the SBI. Although the applicant’s
performance rating is not considered in the SIDP
as part of the decision to refer to the SBI, it may
contain information that is redundant with peer
and supervisory ratings of the applicant on the
PSA dimensions. In fact, performance ratings did
show saall but cignificant correlations with esch
of the PSA dimension ratings. Thus, the perfor-
mance rating may reflect similar or additional
information on the differences among applicants
in supervisory and technical abilities.

The two groups considered in the analysis of
promotability were not significantly different on
the PSA Composite or technical performance
variables. Of the independent variables used in
the logistic regression analysis, only the SBI
composite predicted promotability. This is
perhaps as it should be, since the SBI is meant to
identify more keenly the skills rated by
supervisors and peers in the PSA {excluding
technical competence) and additional important
competencies as identified in a study of FAA

air traffic control first-line supervisor
competencies (Myers & Stutzman, 1993).
Because the SBI is conducted with interviewers
who do not know the candidate and the content
of the scenarios does not include aspects of the
air traffic controller’s operational job, technical
performance (as measured by performance
ratings and experience) should not play a role in
performance at this stage. Of course, an
individual’s performance in the SBI may be
enhanced through other types of work
experience, for exampie, prior staff positions or
experience as an on-the-job-training or Academy
instructor. These types of experiences bring with
them opportunities to interact with others to
accomplish tasks unlike those normally found in
the air traffic controller’s operational position,
and can enhance current supervisory skills and
the development of new skills.

Differences between those who were selected
for a supervisory position and those who were
not were found for the PSA Composite, SBI
Knowledge Composite, and Performance Rating
variables, Although we cannot know for certain
which factors the selecting official is emphasizing
in making a selection decision based on this
analysis, it does appear that several factors are
being considered in the selection of supervisors.
The measures for which there were significant
mean differences encompassed the 5 most
frequently mentioned competencies important to
task accomplishment (communication, leadership,
and interpersonal skills; knowledge of
supervisory role performance. and technical
competence) identified by incumbent air traffic
contre: supervisors (Myers & Stutzman, 1991).
Thus, selection officials appear to be choosing
individuals for supervisory positions in line with
incumbent perceptions of important competencies
in performing the supervisory job. Given the
various sources of the ratings (peers, supervisors,
and trained assessors) selecting officials may also
be considering multiple inputs on the applicant’s
qualifications in making the selection decision.
However, these results must be considered in
light of the fact that only the PSA Composite
entered into the logistic regression equation to
predict selection status and the model did not fit
the data well. The importance of different factors
in attaining a supervisory position is stil] not well


¢ dan

Alta rala fa (o nH (a IH(v SLISOUV (Z

9x (a gala ax (lO Ogl(a XaAlv xaxaxox (1
$1919]

12

iWwaaein\ [eva]Ke

saavo POs\eaal ,

sjoqu kg
ONINOSVAN LOVULSAV