DOT/FAA/AM-98/22

Dana Broach

Civil Aeromedical Institute

Oklahoma City, Oklahoma 73125

August 1998

Final Report

This document is available to the public

through the National Technical Information

Service, Springfi eld, Virginia 22161.

 

Offi ce of Aviation Medicine

Washington, DC 20591

Summative Evaluation of the 

Collegiate Training Initiative 

for Air Traffic Control Specialists 

Program: Progress of Minnesota 

Air Traffic Control Training Center 

Graduates in En Route Field Training

U.S. Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
Technical Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. 

 DOT/FAA/AM-98/22 

4. Title and Subtitle 

Summative Evaluation of the Collegiate Training Initiative

5. Report Date

August 1998 

for Air Traffic Control Specialists Program: 

Progress of Minnesota Air Traffic Control Training Center 

Graduates in En Route Field Training

6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No. 

 Broach, D. 

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

 FAA Civil Aeromedical Institute 

 P.O. Box 25082

 Oklahoma City, OK 73125 11. Contract or Grant No.

12. Sponsoring Agency Name and Address 

 Office of Aviation Medicine 

 Federal Aviation Administration 

 800 Independence Avenue, S. W. 

 Washington, DC 20591

13. Type of Report and Period Covered

15. Supplementary Notes

This research was conducted under task AAM-97-B-HRR-133

16. Abstract

This summative evaluation of the Collegiate Training Initiative for Air Traffic Control Specialists focused on the 

progress of the Minnesota Air Traffic Control Training Center (MnATCTC) graduates in en route field training. 

The evaluation compared 136 MnATCTC graduates with 157 FAA Academy graduates on 4 classes of measures: 

(a) diversity; (b) progress in training at the first assigned field facility; (c) attrition from the first assigned field 

facility; and (d) performance ratings at the first assigned facility. A cost-benefit analysis for the MnATCTC program 

was also conducted. There were significantly more women in the MnATCTC (40%) than in the FAA Academy 

group (17%); there were no significant differences in minority representation. Just 17% of the MnATCTC had 

achieved full performance level (FPL) certification as of June 1995, compared with 69% of the FAA Academy 

group. However, time to FPL and attrition rates were similar. MnATCTC graduates were rated significantly lower 

than FAA Academy graduates by supervisors in teamwork, technical skill, technical knowledge, and overall potential 

to succeed in the ATCS occupation. Cost analysis found that MnATCTC per-hire costs would be competitive with 

FAA Academy costs-per-student by FY1998–2000. Cost-benefit analysis found that the MnATCTC would begin 

returning about $1.45 in avoided costs and savings to the agency for every $1 invested by FY1998–2001, even with 

continued FAA financial support. However, with a maximum capacity of about 100 graduates per year, the 

MnATCTC can provide only a small fraction of the FAA controller workforce. In summary, this evaluation found 

that the MnATCTC program appears to be achieving its stated goals. 

17. Key Words

 Air Traffic Control Specialist (ATCS) 

 Training, Program Evaluation, Education 

 Cost-Benefit Analysis

18. Distribution Statement

 Document is available to the public 

 through the National Technical 

 Information Service, Springfield, VA 22161

19. Security Classif. (of this report)

 UNCLASSIFIED

20. Security Classif. (of this page)

 UNCLASSIFIED

21. No. of Pages

76

22. Price

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

i

iii

TABLE OF CONTENTS

Page

INTRODUCTION .......................................................................................................................................... 1

Program implementation .............................................................................................................................. 1

Evaluation design .......................................................................................................................................... 2

METHOD ........................................................................................................................................................ 3

Sample .......................................................................................................................................................... 3

Measures........................................................................................................................................................ 4

Training progress ................................................................................................................................. 4

Performance ratings ............................................................................................................................ 5

Benefits and costs ................................................................................................................................. 5

Procedure ...................................................................................................................................................... 6

Field training progress data collection ..................................................................................................... 6

Attrition data collection ........................................................................................................................... 6

Performance ratings data collection .......................................................................................................10

Survey administration ........................................................................................................................10

Survey response rates..........................................................................................................................11

Respondent characteristics .................................................................................................................11

Costs and benefits ..................................................................................................................................12

Framework ..........................................................................................................................................12

Costs ...................................................................................................................................................12

Benefits ...............................................................................................................................................12

Analyses .......................................................................................................................................................13

Employee diversity .................................................................................................................................13

Field training progress ............................................................................................................................13

Attrition ..................................................................................................................................................13

Performance ratings ................................................................................................................................13

Costs and benefits ..................................................................................................................................14

RESULTS ........................................................................................................................................................14

Employee diversity ...................................................................................................................................... 14

Field training progress ................................................................................................................................14

Training outcomes ...................................................................................................................................... 17

Attrition ..................................................................................................................................................17

Survival analysis...................................................................................................................................... 17

FPL certification .........................................................................................................................................21
iv

Time to FPL ...........................................................................................................................................21

Accession to FPL.........................................................................................................................................21

Performance ratings .................................................................................................................................... 21

Reliability................................................................................................................................................21

OJT Instructor........................................................................................................................................22

Controller ...............................................................................................................................................24

Supervisor ...............................................................................................................................................25

Influence of degree of acceptance and delay on supervisor ratings.......................................................25

Degree of acceptance..........................................................................................................................25

Hiring delay........................................................................................................................................28

Joint effects of acceptance and hiring delay ......................................................................................28

Costs and benefits .......................................................................................................................................31

Costs .......................................................................................................................................................31

Program costs..........................................................................................................................................31

Cost comparison .....................................................................................................................................31

Benefits........................................................................................................................................................33

Program benefits.....................................................................................................................................33

Cost-benefit analysis...............................................................................................................................33

DISCUSSION ................................................................................................................................................38

REFERENCES ................................................................................................................................................40

APPENDIX A: Performance rating instruments ......................................................................................... A-1

OJT Instructor rating instrument ............................................................................................................ A-3

Controller rating instrument .................................................................................................................... A-8

Supervisor rating instrument .................................................................................................................. A-13

APPENDIX B: Cost-benefit analysis worksheets ........................................................................................ B-1

LIST OF FIGURES

Figure Page

1 MnATCTC and FAA Academy attrition/survival analysis .......................................................................20

2 MnATCTC and FAA Academy accession to FPL over time analysis .......................................................22

3 MnATCTC projected cost-per-hire ...........................................................................................................35

4 MnATCTC cumulative costs and benefits ................................................................................................38

5 MnATCTC cumulative benefit-to-cost ratio ............................................................................................39
v

LIST OF TABLES

Table Page

1 CTI/ATCS hiring by fiscal year and participating institution ............................................................... 2

2 Air Route Traffic Control Center (ARTCC) assignments by program .................................................. 3

3 Facility reporting dates by program ......................................................................................................... 4

4 Minority status and gender ...................................................................................................................... 4

5 Representation of equal employment opportunity (EEO) groups.......................................................... 5

6 En route training curriculum ................................................................................................................... 7

7 En route OJT curriculum track followed by group ..............................................................................10

8 Performance evaluation survey return rates ...........................................................................................11

9 Demographic characteristics of controller survey respondents .............................................................11

10 En route days in phase, hours OJT in phase, and phase Indicator of Performance for

MnATCTC and FAA Academy graduates.............................................................................................15

11 Standardized en route days in phase, hours OJT in phase, and phase Indicator of

Performance for MnATCTC and FAA Academy graduates .................................................................18

12 Outcomes at first assigned en route facility ..........................................................................................20

13 Years to full performance level (FPL) at first assigned en route facility ...............................................21

14 Performance ratings inter-rater correlations by rating source...............................................................23

15 Analysis of OJT-I performance ratings by group ..................................................................................24

16 Analysis of controller performance self-ratings by group ......................................................................24

17 Analysis of supervisor performance ratings by group ............................................................................25

18 Analysis of supervisor performance ratings by group, controlling for controller’s perceived

degree of acceptance in the facility ........................................................................................................26

19 Analysis of supervisor performance ratings by group, controlling for delay between graduation

and hiring ...............................................................................................................................................29

20 Supervisor ratings of teamwork, controlling for ATCS’s perceived degree of acceptance in the

facility and delay between graduation and hiring .................................................................................30

21 Supervisor ratings of technical skill, controlling for ATCS’s perceived degree of acceptance

in the facility and delay between graduation and hiring .......................................................................30

22 Supervisor ratings of technical knowledge, controlling for ATCS’s perceived degree of

acceptance in the facility and delay between graduation and hiring ....................................................32

23 Supervisor ratings of potential to succeed in the ATCS occupation, controlling for the

ATCS’s perceived degree of acceptance in the facility and delay between graduation and hiring .............32

24 Annual FY costs worksheet logic ...........................................................................................................34

25 Cost-benefit worksheet logic ..................................................................................................................36

vii

EXECUTIVE SUMMARY

Overview

This first summative evaluation of the Collegiate

Training Initiative for Air Traffic Control Specialists

(CTI/ATCS) focused on the progress of Minnesota

Air Traffic Control Training Center (MnATCTC)

graduates in en route field training. The evaluation

was framed by two questions:

• How are MnATCTC program graduates progressing in en route field training relative to a comparison

group of FAA Academy graduates?

• What are the costs and benefits of the MnATCTC

program for the Federal Aviation Administration

(FAA)?

Method

The evaluation compared 136 MnATCTC graduates hired between 1990 and 1993 with 157 FAA

Academy graduates who entered en route field training in May and June 1991. The groups were compared on 4 measures:

• Diversity

• Progress in en route field training at first assigned en

route facilities

• Attrition from first assigned en route facilities

• Performance ratings at first assigned en route

facilities

Supervisory, on-the-job training instructor (OJTI), and self-evaluations were collected in spring 1994

by mail. Field training and attrition measures were

extracted from FAA and Civil Aeromedical Institute

(CAMI) databases in June 1995.

Direct and indirect costs associated with the

MnATCTC program were estimated and projected

through the year 2003 to compare per-hire costs with

FAA Academy per-controller costs. Four classes of

monetary benefits to the agency were identified: (1)

avoided screening costs; (2) avoided Academy training costs; (3) avoided performance verification (PV)

costs; and (4) savings from reduced time-to-FPL for

MnATCTC graduates. Program benefits were estimated and projected through the year 2003 to conduct a benefit-to-cost analysis for the MnATCTC

program as a model for future CTI/ATCS costbenefit analyses.

Results

Diversity

There were significantly more women in the

MnATCTC group (40%) than in the FAA Academy

sample (17%). There was no difference in the representation of minorities.

Field training progress

Training phases. MnATCTC graduates did better

on six measures of training performance, relative to

historical facility averages: number of days to complete PHASE VII; number of days to complete PHASE

IX (raw and adjusted for the number of sectors for

which training was provided); number of days in

PHASE X (raw and adjusted for number of sectors for

which training was provided); and number of days in

PHASE XI. FAA Academy graduates did better than

MnATCTC graduates on three measures of training

performance, relative to historical facility averages:

indicator of performance (IP) ratings in PHASE VI;

number of days in PHASE VIII; and number of hours

of on-the-job training (OJT), adjusted for number of

sectors, in PHASE X.

Time to FPL. Just 23 of the MnATCTC graduates

(17%) had reached FPL at the first assigned field

facility as of June 1995, in contrast to 108 of the FAA

Academy graduates (68.8%). There was no significant difference in the average times to FPL between

the groups. Survival analysis found no significant

differences between the groups in accession to FPL

over time.
viii

Attrition

Ten developmentals had attrited at the first assigned facility from the MnATCTC group, compared with six FAA Academy graduates. The attrition

rates were not statistically different. Significantly

more MnATCTC graduates (N = 7) had changed

facilities or options than FAA Academy graduates (N =

1; Z = 1.99, p < .05). No significant differences

between groups in attrition over time were found by

survival analysis.

Performance ratings

Supervisors rated controllers on items representing five dimensions: (a) technical skill; (b) technical

knowledge; (c) teamwork; (d) degree to which the

developmental was accepted by others in the facility;

and (e) global assessment of potential to succeed in

the ATCS occupation. MnATCTC graduates were

rated significantly lower than FAA Academy graduates on all five dimensions. In particular, the average

rating on potential to succeed in the occupation of 81

for MnATCTC graduates was significantly lower than

the average rating of 86 for FAA Academy graduates

(on a 40 to 100 scale).

Controllers completed a self-evaluation using the

same instrument as supervisors. MnATCTC graduates rated themselves significantly lower on teamwork, technical skill, technical knowledge, and

potential to succeed in the ATCS occupation than

did FAA Academy graduates. MnATCTC graduates

also felt significantly less well accepted at the facility

than FAA Academy graduates.

Program costs and benefits

The comparison of MnATCTC per-hire costs

with FAA Academy per controller costs found that

the MnATCTC per-hire costs would be equal to or

less than FAA Academy per controller costs by the

FY1998-2000 timeframe.

The benefit-to-cost analysis found that the

MnATCTC program would begin returning a dollar

or more for every dollar invested in the program by

the FY1998-2000 timeframe, given current hiring

projections, even with continued FAA financial support of about $1.5 million per year to produce 100

new graduates each year.

Capacity relative to projected demand

The MnATCTC has a maximum capacity, according to Congressional testimony by the program director, of producing 100 graduates per year with a

continuing financial subsidy from the FAA. Analysis

of current ATCS workforce demographics suggests a

much larger hiring demand after the turn of the

century. It is unlikely, therefore, that the MnATCTC

can supply more than a small fraction of FAA

workforce requirements.

Conclusions

Overall, the MnATCTC program appears to be

meeting most of its stated objectives. The MnATCTC

is contributing to agency diversity goals for women,

but not minorities. The progress of MnATCTC

graduates through field training is essentially no

different than the progress of FAA Academy graduates. MnATCTC were rated significantly lower on

five job performance dimensions than FAA Academy

graduates. MnATCTC appears to be able to produce

new controllers on a cost-competitive basis with the

FAA Academy. FAA benefits accruing largely from

avoided FAA Academy training costs are not likely to

balance cumulative program costs in the near-term

(FY1997-2000). Over the longer term, however, accrued

FAA benefits may be greater than cumulative costs, even

with continued FAA financial support. Alternative approaches that require little or no agency investment in

and support of external training infrastructure at the

cost of internal training capacity may yield greater ratesof-return, as well as satisfy larger proportions of the

agency’s future workforce requirements.
1

SUMMATIVE EVALUATION OF THE COLLEGIATE TRAINING INITIATIVE

FOR AIR TRAFFIC CONTROL SPECIALISTS PROGRAM:

PROGRESS OF MINNESOTA AIR TRAFFIC CONTROL TRAINING CENTER

GRADUATES IN EN ROUTE FIELD TRAINING

INTRODUCTION

Comprehensive reviews of the Federal Aviation

Administration (FAA) air traffic control specialist

(ATCS) recruiting, selection, and training programs

completed in the late 1980s (Schultz & MarshallMies, 1988; Means et al., 1988; Northern NEF, Inc.,

1988) recommended that the agency rely more heavily

on the technical training expertise available through

two- and four-year colleges and universities. In response to those studies, the FAA developed the Flight

Plan for Training (Flight Plan; Office of Training and

Higher Education, 1989) in which the development

of pre-hire controller training at the college and/or

university level was identified as a specific initiative

(p. 28):

Pre-hire training at the college and university level

will provide an economical new source of highly qualified and motivated Air Traffic Control Specialists. To

test this concept, a trial Air Traffic Control (ATC)

training program will be conducted. A university-based

pilot program in which one hundred controllers will

earn undergraduate degrees and receive the equivalent

of the FAA Academy’s developmental training will be

initiated in 1989. If hired by the agency, these controllers will enter the FAA training system at an advanced

level. The agency will expand this program to other

universities and colleges if the experimental program is

successful.

Program implementation

Implementation of this concept began in 1989,

with approval and funding for two programs: the

Minnesota Air Traffic Control Training Center

(MnATCTC), as administered by the Mid-America

Aviation Resource Consortium, Eden Prairie, Minnesota; and Hampton University, Hampton, Virginia. The program was subsequently expanded under

the direction of the Higher Education and Advanced

Technology Staff (AHT-30), culminating in the 1991

FAA Order 3120.26. The order formally established

the Collegiate Initiative for Air Traffic Control Specialists (CTI/ATCS) as a test program and provided

for the selection of participating educational institutions. By 1992, five institutions were participating in

the CTI/ATCS program:

•The Minnesota Air Traffic Control Training Center (MnATCTC), Eden Prairie, Minnesota, as

administered by the Mid-America Aviation Resource Consortium (MARC);

• Hampton University (HU), Hampton, Virginia;

•Community College of Beaver County (CCBC),

Monaca, Pennsylvania;

• University of North Dakota (UND), Grand Forks,

North Dakota; and

• University of Alaska at Anchorage (UAA), Anchorage, Alaska.

However, full implementation of the program

coincided with a significant downturn in ATCS

hiring requirements in 1992 and 1993 (Morrison,

Fotohui, & Broach, 1996), resulting in a lower rate of

hiring of CTI/ATCS program graduates than had

been expected. The Office of the Inspector General

for the Department of Transportation (DOT/IG)

conducted an audit of FAA higher education programs at about the same time. That DOT/IG (1993,

p. 14-15) recommended that the FAA discontinue

future funding for the CTI/ATCS program and advise the appropriate congressional committees of the

program’s limited success, absence of hiring opportunities for program graduates, and intention to

recommend discontinuance of the program. However, the FAA formally disagreed with the DOT/IG

recommendations on the CTI/ATCS program, pending an evaluation of the long-range recruitment need

for air traffic control specialists. Moreover, at the

time of the DOT/IG audit, only a few (61) CTI/

ATCS program graduates had been hired by the FAA

from the MnATCTC program. Since that date, the
2

number of graduates from MnATCTC and other

participating institutions hired by the FAA has increased substantially (Table 1), providing a larger

sample to support an empirically based summative

evaluation. As shown in Table 1, the largest number

of CTI/ATCS graduates hired by the FAA has come

from the MnATCTC program. In addition, the

MnATCTC program has been a subject in budget

negotiations between the Congress and the FAA since

FY94. Therefore, this report focuses specifically on

the MnATCTC program in an empirical evaluation

of goal attainment relative to agency hiring requirements, costs, and benefits.

Evaluation design

Evaluations can be broadly categorized into three

classes, based on the primary focus of the research: (a)

program conceptualization; (b) program implementation; and (c) program utility (Rossi & Freeman, 1985).

The formative evaluation conducted in fiscal year 1993

(Morrison, Fotohui, & Broach, 1996) concentrated on

program implementation at each of the five participating educational institutions. That evaluation specifically addressed the degree of innovation that the five

participating institutions demonstrated with regard to

their (a) recruiting activities, (b) selection procedures,

and (c) training methods. Overall, the participating

institutions, including MnATCTC, had developed recruiting, selection, and training methodologies and

technologies that differed substantially from those used

by the FAA. This summative evaluation addresses outcomes, with particular attention to the MnATCTC

program. MnATCTC graduate diversity and performance in field facility training is compared with that of

a control group of FAA Academy graduates.

It should be noted that the control group in this

evaluation is not strictly equivalent to the “experimental” group of MnATCTC graduates. For example, the control group is not drawn from the same

en route facilities to which MnATCTC graduates

were assigned (Table 2). A third of the MnATCTC

graduates were assigned to the Minneapolis Air Route

Traffic Control Center (ARTCC); another 20% were

assigned to the Oakland ARTCC. In contrast, the

control group of FAA Academy graduates were scattered across 20 of 23 en route centers (excluding

Guam). Nor is the control group matched to the

MnATCTC graduates in terms of hiring dates (Table

3). The control group entered field training in mid1991, while the MnATCTC graduates entered service from late 1991 through 1993. As a consequence,

the control group had been at the first assigned en

route facility an average of 38 months (SD = 11.3

months), compared with 28 months (SD = 9.32

months) for the MnATCTC graduates (F (1,291) =

71.80, p < .001), as of June 1995. Therefore, the

research design can be characterized as a post-treatment comparison with a non-equivalent control

group, which is considered a relatively “weak” design

in terms of controls for validity and generalizability

(Campbell & Cook, 1976). However, this design

does provide at least some objective data as the basis

Table 1. CTI/ATCS Hiring by fiscal year and participating institutiona

Participating Institution

FY MnATCTC HU CCBC UND UAA Total

88-90b 3 3

91 26 10 36

92 19 2 21

93 78 18 96

94 40 1 0 7 5 53

95 30 14 35 1 80

96 17 6 6 29

Total 193 32 74 7 12 318

Notes: a

Data provided by Aviation Careers Examining Division (AMH-300)

b

FY88-90 CCBC Cooperative Education Program graduates counted as

CTI/ATCS hires
3

for assessing MnATCTC program goal attainment,

costs, and benefits relative to the known standard of

FAA Academy graduates.

Goals, costs, and benefits should be operationally

defined in evaluation research such as this study

(Bloom, 1967; Rossi & Freeman, 1985). Specific

operationally-defined criteria for the summative evaluation of the MnATCTC program relative to traditionally hired and trained FAA Academy graduates

included (a) increased employee diversity, (b) improved employee performance as represented by instructor, self, and supervisor evaluations, (c) reduction

in time for graduates of these programs to complete

the ATCS field training sequence, and (d) reduction

of attrition during field training among MnATCTC

graduates. The latter two operational criteria, along

with specific incurred and avoided costs associated

with the program, provide a basis for assessing the

relative benefits of the initiative.

METHOD

Sample

Two groups of controllers were identified as the

sample for this evaluation: 136 MnATCTC graduates hired by the FAA between 1991 and 1993 (see

Table 3 for hiring dates), and the 157 FAA Academy

graduates (“FAA Academy”) that entered en route

ATCS field training in May and June 1991. The FAA

Academy graduates included persons who had participated in the development and validation of the

FAA’s ATCS Pre-Training Screen (Broach & BrechtClark, 1994). Facility assignments and hiring dates

by program are presented in Tables 2 and 3. The overall

demographic characteristics by program are described

in Tables 4 (minority status and gender) and 5 (Equal

Employment Opportunity (EEO) subgroups).

Table 2. Air Route Traffic Control Center (ARTCC) assignments by program

MnATCTC FAA Academy

FACID Name N % N %

ZAB Albuquerque 9 5.7

ZAN Anchorage 7 5.1

ZAU Chicago 10 6.4

ZBW Boston 2 1.3

ZDC Washington 1 0.7 8 5.1

ZDV Denver 11 8.1

ZFW Fort Worth 5 3.7 12 7.6

ZHN Honolulu 2 1.3

ZHU Houston 11 7.0

ZID Indianapolis 5 3.7 7 4.5

ZJX Jacksonville 1 0.7 8 5.1

ZKC Kansas City 18 13.2

ZLA Los Angeles 2 1.9 3 1.5

ZLC Salt Lake City 1 0.7 7 4.5

ZMA Miami 15 9.6

ZME Memphis 3 2.2 16 10.2

ZMP Minneapolis 46 33.8 11 7.0

ZNY New York City 2 1.5 12 7.6

ZOA Oakland 1 0.6 27 19.9

ZOB Cleveland 11 7.0

ZSE Seattle 9 5.7

ZSU San Juan CERAP 1 0.7

ZTL Atlanta 6 4.4 3 1.9
4

Table 3. Facility reporting dates by program

MnATCTC FAA Academy

Reporting Date N % N %

May 1991 72 45.9

June 1991 83 52.9

September 1991 15 11.0 2 1.2

November 1991 11 8.1

March 1992 16 11.8

April 1992 1 0.7

May 1992 3 2.2

November 1992 2 1.4

January 1993 27 19.9

February 1993 3 2.2

March 1993 11 8.1

April 1993 2 1.5

May 1993 12 8.8

June 1993 11 8.0

July 1993 2 1.5

October 1993 1 0.7

November 1993 10 7.3

December 1993 6 4.4

January 1994 3 2.2

Table 4. Minority status and gender of evaluation sample

MnATCTC FAA Academy Total

Characteristic N % N % N %

Minority status

Nonminority 111 81.4% 136 86.6% 247 84.3%

Minority 12 8.8% 21 13.4% 33 11.3%

Missing 13 3.7% 13 4.4%

Gender

Male 82 60.3% 130 82.8% 212 72.4%

Female 54 39.7% 27 17.2% 81 27.6%

Missing

Measures

Dependent variables, representing the criteria for

this summative evaluation, included (a) representation by EEO group, (b) rate of progress in field

training, (c) attrition from en route field training, (d)

on-the-job training instructor (OJTI), self, and supervisor ratings of field performance, and (e)

MnATCTC program costs and benefits.

Training progress

One means of achieving the goal of safe, orderly,

and efficient operation of the National Airspace

System is to ensure that the persons directing air

traffic have reached the full performance level (FPL).

FPL controllers are the backbone of the controller

workforce, as they are responsible on a moment-tomoment basis for the safe, orderly, and expeditious
5

flow of air traffic through the National Airspace

System. Development of FPL skills and knowledge

requires extended, intensive, and expensive formal

and on-the-job training for trainee (“developmental”) controllers. The training averages three years in

en route centers (Manning, Della Rocco, & Bryant,

1989). This extended field training represents a significant agency expense directly proportionate to the

time spent in training. For example, the Air Traffic

Training Work Group (ATTWG; 1991a) in a comprehensive review of ATCS training programs, estimated field training costs at about $131,739 per en

route controller over a 36-month interval. Innovative

programs such as those offered at MnATCTC have

the potential to reduce overall training times, thereby

reducing the agency’s costs. Similarly, attrition from

the field training sequence represents a lost investment for the agency. Reductions in attrition rates,

therefore, translate directly into cost savings.

Performance ratings

Evaluations of technical skill, knowledge, and

teamwork provide additional information about core

job performance not available in the training progress

or attrition measures. Core technical job performance may be thought of as the product of knowledge, skill, and motivation (Campbell, 1990;

Campbell, McCloy, Oppler, & Sager, 1993). Items

to assess individual technical knowledge and skill

were adapted directly from the existing instruments

used for over-the-shoulder evaluation of technical

performance (FAA Form 3120-25). Teamwork is a

dimension of controller training and job performance that has become a recent focus of concern (Air

Traffic Training Workgroup, 1991a,b; E. L. Hamm

& Associates, Inc., & Hampton University, 1990;

Hartel & Hartel, 1995; Newcomb & Jerome, 1994;

Seamster, Cannon, Pierce, & Redding, 1992; Sherman

& Helmreich, 1993). For example, teamwork was

incorporated as an explicit evaluation dimension in

the post-training performance verification process

(Performance Verification Division, undated). Items

representing teamwork in the evaluation instrument

were based on a review of the teamwork and interpersonal skills literature with reference to air traffic

control job analysis information (Stark, 1994).

In addition, concerns were raised by participating

institutions that the degree to which a CTI graduate

was accepted in the facility by co-workers, instructors, and management might influence OJT assignments, training, and perceptions of performance.

Items were included, therefore, in the performance

rating, to represent the degree to which the ratee felt

or was perceived as being accepted in the workplace.

Finally, a global subjective assessment of the ratee’s

potential to succeed in the ATCS occupation was

incorporated into the performance rating instrument.

Three parallel versions of the instrument were developed for first line supervisors, OJT instructors, and

incumbent controllers. A sample of the OJT instructor, controller, and supervisor evaluation forms and

associated cover letters are provided in Appendix A.

Benefits and costs

The training and survey data described above

provide information about the degree to which the

MnATCTC is meeting its program objectives, in

terms of the progress and performance of program

graduates, in comparison with controllers entering

the occupation through the FAA Academy program.

While knowledge, both of the manner in which a

program such as the Minnesota program has been

Table 5. Representation of equal employment opportunity (EEO) groups

MnATCTC FAA Academy Total

Characteristic N % N % N %

Nonminority Male 66 48.5% 111 70.7% 177 60.4%

Nonminority

Female

45 33.1% 25 15.9% 70 23.9%

Minority Male 9 6.6% 19 12.1% 28 9.6^

Minority Female 3 2.2% 2 1.3% 5 1.7%

Missing 13 9.6% 13 4.4%
6

implemented, and of its outcomes, is indispensable

to program managers, stakeholders, and policymakers,

it is just as critical in the evaluation process to provide

information about costs relative to benefits (Rossi &

Freeman, 1985). Inputs to the program and outcomes are measured in monetary terms in cost-benefit analyses to support program evaluation. Therefore,

the final step in the evaluation was to develop a

framework for measuring the costs associated with

and benefits accruing from the CTI/ATCS program.

That framework was then applied to the evaluation of

the MnATCTC program as a model for future costbenefit evaluations for each institution participating

in the overall CTI/ATCS program.

Procedure

Field training progress data collection

Field training data for both samples were extracted

from the Civil Aeromedical Institute ATCS Training

Tracking (TRACKING) database. This database was

maintained under the FAA ATCS National Training

Tracking Program order (FAA Order 3120.22A; FAA,

1985) through June 1995. The phases of training are

described in Table 6, based on the 1988 En Route

Instructional Program Guide (IPG; FAA, 1988).

There are variations allowed in the training phases,

based on local facility requirements, as noted in

Table 6. For example, PHASE VII (PRELIMINARY RADAR-ASSOCIATED/NONRADAR CONTROL TRAINING AND

ASSISTANT CONTROLLER DUTIES) is conducted in the

classroom for up to 8 weeks (40 days). The variation

named PHASE VIIB adds 15 nonradar familiarization

problems in the facility dynamic simulation laboratory (“DYSIM”) to the classroom instruction. Other

phases with variations include PHASE VIII (3 variations, differing in the total number and mix of

nonradar and radar problems run in the DYSIM),

PHASE XI (2 variations, differing in the total number

of hours and DYSIM radar problems), and PHASE

XIII (2 variations, differing in the total number of

OJT hours allowed per sector). The sequences of

phases and their variations (known as a “tracks” or

curricula) taken by each controller, are summarized

in Table 7. The majority of both MnATCTC and

FAA Academy graduates followed either what is

known as the “B-track” (substituting phases VIIB,

VIIIB, XIB, and XIIIB for the standard phases VII,

VIII, XI, and XIII), or what is called the “AB Standard” track, with phases VIIIA and XIB in place of

the standard phase VIII and XI. A majority of the

MnATCTC (53.7%) graduates followed the ABStandard track, with a substantial minority (23.5%)

following the B-track. In comparison, a plurality of

FAA Academy graduates (42.0%) pursued the BTrack, with another 42.7% following the AB-Standard curriculum. Overall, despite the differences in

facility assignments, the curricular tracks followed by

graduates from the two programs (Academy,

MnATCTC) are reasonably similar.

Training data available included the number of

days spent in each phase of training, hours of on-thejob training (OJT) taken in each phase, and the

overall rating of performance in that phase on a 1

(lowest 10% of controllers observed) to 6 (top 10%

of all controllers observed) scale. The availability of

data for subjects in each phase of training was entirely

dependent upon the timeliness of reporting facilities;

all data, as reported by the facilities, were extracted

for this analysis. The reliability of the ATCS training

tracking data has been described by Manning (1990).

Incoming data were closely examined by CAMI research technicians for out-of-range and missing values; follow-up calls were made to the reporting facility

as required to verify and complete data. The timebased measures (days in phase; hours of OJT) are

ratio scale variables; any unreliability in those measures would be due to errors in reporting at the

facility level, or to clerical errors in data handling.

The reliability of the subjective assessment of developmental performance in a phase of training (the IP)

is undocumented. However, studies of performance

ratings in the literature suggest that such ratings are

reasonably reliable and useful for a variety or workforce

research purposes (Borman, White, Pulakos, &

Oppler, 1991; Hoffman, Nathan, & Holden, 1991;

Smith, 1976).

Attrition data collection

Data to identify attritions from training for both

samples were extracted from the FAA Consolidated

Personnel Management Information System (CPMIS)

and cross-referenced with the CAMI ATCS Training

Tracking (TRACKING) database. There is no single

field in either data source indicating a training attrition; rather, multiple data elements from both sources

must be evaluated and combined to determine the

outcome for a given case. Outcome coding for this

analysis was based on CPMIS and TRACKING data

fields representing (a) grade level, (b) training phase
7

Table 6. En route training curriculum

Purpose Length Environment Topics

PHASE V: Assistant Controller Training

To prepare developmental forassistant controller positionqualification and certification

40 days Classroom Center, area, and sector charts

Flight data processing, including computer (HOST) messageentriesInterphone (land line) operation

PHASE VI: Assistant Controller Position Qualification/Certification

Qualify developmental to performfull range of assistant controllerduties and to obtain certification onall assistant controller positions ofoperation in an assigned area ofspecialization

80 hoursper area

OJT Receive, process, coordinate, and deliver flight plan

information

PHASE VII (VIIB): Preliminary Radar-Associated/ Nonradar Control Training and Assistant Controller Duties

To provide background knowledgein preparation for entry into radarassociated/ nonradar training

80 hours(15 LabsVIIIB)

ClassroomDYSIM

Detailed area and sector chartSpecial military operationsLetters of Agreement/facility ordersPhraseology and strip-marking(Nonradar familiarization problems)

(Table 6 continues)
8

(Table 6 continued)

Purpose Length Environment Topics

PHASE VIII (VIIIA, VIIIB): Radar-Associated/ Nonradar Controller Training

To prepare developmental forinitial radar-associated/ nonradarcontrol position qualification andcertification

56 hours33 Labs(21 Labs -VIIIA)(42 Labs -VIIIB)

ClassroomDYSIM

Strip-markingInterphone/radio phraseology and proceduresIssue IFR and other clearances to provide vertical,longitudinal, or lateral separation according to priorityto departing, arriving, and holding aircraftPosition relief briefingRadar identification procedures and radar separationminimaApply procedures for verifying and using Mode CApply transition procedures to and from primary backup system

PHASE IX: Initial Radar-Associated/ Nonradar Control Position Qualification and Certification

To qualify developmental toperform full range of duties andattain certification on 2 radarassociated/ nonradar controlpositions of operation in an areaof specialization

180 hoursper position(90 daysmax)

OJT Initiate and accept radar handoffs and pointouts

Perform appropriate changeover procedures totransition to and from primary back-up systemMaintain separation using prescribed standardsIssue departure clearances, beacon codes, holdingprocedures, weather advisoriesEmergency procedures: inflight emergencies, radiocommunication failure, hijackingsSector and board management

(Table 6 continues)
9

(Table 6 continued)

Purpose Length Environment Topics

PHASE XIII (XIIIB): Final Radar Control Position Qualification and Certification

To qualify developmental toperform full range of duties andattain certification on allremaining radar positions ofoperation (sectors) in an area ofspecialization

120 hourspersector,120 daysmax; or300 hourspersector,300 daysmax(XIIIB)

OJT Application and use of all topics and skills covered in

previous phases of training under supervision of OJTInstructor(s) with live traffic

Source: FAA, 1988
10

completions, (c) training phase grades, (d) facility

type and level, (e) facility disposition codes, and (f)

facility types and levels at time of entry into field

training and at time of data extraction. Possible

training outcomes at the first assigned field facility

included: (1) separation from the ATCS occupation

(which may or may not involve termination from

employment by the FAA); (2) attrition (without

separation from the FG-2152 occupation or agency)

from the first en route facility through facility or

option change (e.g., switch to terminal or flight

service); (3) still in developmental (training) status as

a controller at the first en route facility; and (4)

achieved Full Performance Level (FPL) in the ATCS

occupation as of June 1995 at the first en route

facility. Only outcomes at the first assigned facility

are considered in this analysis for two reasons.

First, the training tracking system was originally

designed to follow progress through OJT to the FPL

at the first facility; the data for second or third

assignments are both less reliable and less complete.

Second, there are significant financial costs associated with moves from the first assigned facility prior

to achievement of the FPL certification. Those costs

may include, but are not limited to, expended training funds, time, permanent change of station (e.g.,

moving) costs, and personnel replacement costs.

Therefore, improved outcomes at the first facility can

lead to significant avoided costs for the FAA.

Performance ratings data collection

Survey administration. Facilities to which

MnATCTC and FAA Academy graduates had been

assigned were identified through CPMIS, as shown

in Table 2. Working through the Air Traffic chain of

command, supervisors and OJT instructors of

MnATCTC and FAA Academy graduates were identified by the facilities. A mail-merge database, linking

controller, supervisor, and OJT instructor, was developed on the basis of the data provided by facilities

to support mailing a ratings package. The ratings

package included a cover letter from the Director of

Air Traffic Program Management (ATZ-1), an explanation of the project, instructions for rating, and the

rating form. The cover letter was addressed to the

rater by name, using the mail-merge database. The

supervisor and OJT instructor forms indicated the

name of the incumbent controller to be rated. Reminder

cards were mailed to controllers, supervisors, and instructors about one month after the initial mailing.

Table 7. Enroute OJT curriculum track followed by group

MnATCTC FAA Academy

Track Phasesa N % N %

Standard V, VI, VII, VIII, IX, X,

XI, XII, XIII

7 5.1

A Track V, VI, VII, VIIA, IX, X,

XI, XII, XIII

11 8.1 5 3.2

B Track V, VI, VIIB, VIIIB, IX, X,

XIB, XII, XIIIB

32 23.5 66 42.0

8B Track: V, VI, VII, VIIIB, IX, X,

XI, XII, XIII

1 0.7 9 5.7

B Variation 1 V, VI, VIIB, VIIIB, IX, X,

XIB, XII, XIII

4 2.9

B Variation 2 V, VI, VII, VIII, IX, X,

XIB, XII, XIII

1 0.7

AB Track V, VI, VII, VIIIA, IX, X,

XIB, XII, XIIIB

7 5.1 10 6.4

Notes: a

Variations in phases for each track shown in boldface.
11

Survey response rates. Performance evaluation

survey response rates are presented in Table 8. OJT

instructors for 53 (39.0%) of the MnATCTC graduates returned surveys, compared with 61 (38.9%)

from OJT instructors of the FAA Academy graduates. The return rates for incumbent controllers from

both groups were better, with 72.1% (98) of the

MnATCTC graduates returning self-evaluation surveys compared with 68.1% (107) of the FAA Academy graduates. Supervisors for 64 of the 136

MnATCTC graduates (47.1%) and 95 of 157 FAA

Academy graduates (60.5%) returned surveys. The

instructor and incumbent return rates for each group

were not statistically different. However, the return

rate for supervisors of the FAA Academy group

(60.5%) was significantly greater than the return

rate for supervisors of the MnATCTC group

(37.2%; Z = -3.25, p < .001).

Respondent characteristics. Demographic characteristics of the controllers that returned performance evaluation surveys are presented in Table 9.

The majority of the MnATCTC graduates returning

surveys were male (72.0%), as were the FAA Academy

graduates (68.5%). All 12 of the minority MnATCTC

graduates, and 20 of 21 minority FAA Academy graduates returned performance self-evaluation surveys.

Table 8. Performance evaluation survey return rates

MnATCTC FAA Academy

OJT Instructors

Mailed 136 157

Total returns 53 61

Return rate 39.0% 38.9%

Controllers

Mailed Controllers 136 157

Total returns 98 107

Return rate 72.1% 68.1%

Supervisors

Mailed Supervisors 136 157

Total returns 64 95

Return rate 47.1% 60.5%***

***p < .001

Table 9. Demographic characteristics of controller survey respondents

MnATCTC Academy

Group NGroup Nrespond %Respond NGroup NRespond %Respond

Gender

Male 82 59 72.0% 130 89 68.5%

Female 54 39 72.2% 27 18 66.7%

Missing

Minority Status

Non-minority 111 77 69.4% 136 87 64.0%

Minority 12 12 100.0% 21 20 95.2%

Missing 13 9 69.2%
12

OJT instructors returning surveys had been controllers for an average of 13 years (SD = 6.7 years).

These instructors had been at their current facility

about 10 years (SD = 5.3 years) and had been in their

current position as an instructor about 8 years on the

average (SD = 5.2 years). They had been providing

training to the developmental being rated an average

of 12.9 months (SD = 24.4 months). There were no

statistically significant differences on these measures

between OJT instructors for MnATCTC and FAA

Academy graduates.

The supervisors who provided performance evaluation ratings on MnATCTC and FAA Academy

graduates for research purposes only in this study

averaged 18.3 years (SD = 6.7) of experience in the

ATCS occupation, with average of 12.5 years (SD =

7.5) in the current facility. These supervisors had an

average of 5.2 years (SD = 3.8) of experience in their

current positions as supervisors and had been supervising the rated CTI/ATCS or Academy graduate an

average of 14.2 months (SD = 10.2). There were no

significant differences between supervisors of

MnATCTC and FAA Academy graduates on the

times as controller, supervisor, or at the current

facility. However, supervisors of Academy graduates

reported significantly longer times supervising the

rated incumbent (M = 15.9, SD = 10.7 months) than

supervisors of MnATCTC graduates (M = 10.7, SD

= 8.0 months; t

(1,199) = 3.36, p < .001). This is

consistent with the previous finding that FAA Academy graduates had been at the first facility longer

than the MnATCTC graduates.

Costs and benefits

Framework. Thompson (1980) identified the following major steps in cost-benefit analysis for program evaluation: (1) identify the decision-maker(s)

who will use the results of the evaluation; (2) identify

alternatives; (3) identify costs; (4) identify benefits;

(5) value program effects in dollars; (6) discount

those values for the effects of time; (7) take distributional effects into account, as appropriate; and (8)

aggregate and interpret the valued effects. This program evaluation is intended to serve the decisionmaking requirements of FAA Air Traffic Services

management. There are two alternatives considered

in the evaluation: training entry-level controllers at

(a) MnATCTC or (b) the FAA Academy. Costs,

benefits, and the valuation of program effects in

dollars are discussed below in detail. Opportunity

costs, time-related discounts in the value of money,

and distributional effects were not addressed in this

initial cost-benefit analysis.

Costs. The agency incurs both direct and indirect

costs in administering the MnATCTC and other

programs under the CTI/ATCS charter. Direct costs

to the agency include the Congressionally-mandated

funds invested in the MnATCTC. Direct costs were

obtained on the basis of public laws passed by the

U.S. Congress (see Morrison, Fotohui, & Broach,

1996, for citations). Indirect costs include (a) program management staff time, (b) evaluation costs,

and (c) other staff costs associated with supporting

the programs. Indirect costs were estimated on the

basis of an electronic mail survey in late 1994 of FAA

program offices and managers. The survey requested

estimates of the proportion of staff time and travel

spent on the program by fiscal year. These indirect

costs were then projected forward to provide estimates through the year 2003.

Benefits. Four major classes of direct, quantifiable

benefits that could be expressed in terms of dollars

were identified for the cost-benefit analysis. First,

under the program structure described in the formative evaluation report (Morrison, Fotohui, & Broach,

1996), MnATCTC graduates took the written ATCS

aptitude test battery but bypassed the screening program at the FAA Academy that was in place through

March of 1992. MnATCTC graduates also bypassed

the successor five-day computerized assessment of

aptitude that was implemented in June 1992 to

replace the former nine-week FAA Academy ATCS

Nonradar Screen (Broach & Brecht-Clark, 1994).

Therefore, the first benefit was avoided screening

costs for MnATCTC graduates. Second, MnATCTC

graduates bypassed the FAA Academy en route training program as implemented in June 1992, and were

placed directly into field training. Therefore, the

second monetary benefit to the agency was avoided

training costs at the FAA Academy. Third, the performance verification (PV) function was delegated to

the MnATCTC from June 1992 onwards; therefore,

the third monetary benefit to the FAA was avoided

costs for PV. Finally, as described in the field training

measures, it was expected that MnATCTC graduates

would take about the same time, or less, than FAA

Academy graduates to attain certification as FPL en

route controllers. Therefore, the fourth monetary

benefit to the agency was savings associated with

reduced times to FPL certification for MnATCTC
13

graduates. Published figures were used to estimate

costs of screening, FAA Academy training, and field

training savings. Cumulative savings were projected

forwards through the year 2003, as with costs.

Analyses

Employee diversity

The representation of women and minorities in

the MnATCTC sample was compared with their

representation in the FAA Academy sample. Fisher’s

Z-test was used to compare the proportions of women

and minorities in the two evaluation samples, on the

hypothesis that:

H1

: Women and minorities will be equally represented in the MnATCTC and FAA Academy samples.

Field training progress

The average number of days, hours of OJT, and

ratings of performance in each phase of the field

training curriculum were compared through a oneway analysis of variance between the two evaluation

samples. However, results based on the raw numbers

of days in phase, hours of OJT, and indicators of

performance can be misleading due to differences in

training programs between air route traffic control

centers (ARTCCs), as well as curriculum differences

noted in Table 7. For example, the historical average

time to complete the same phase of en route training

at one ARTCC may be very different from times at

other centers. These differences may be attributable

to variables such as traffic patterns, facility resources,

and training loads (General Accounting Office,

1989a). As a result, the variability in the raw numbers

of days in phase, hours of OJT, and indicators of

performance may be more attributable to inter-facility differences rather than to between-group differences. To compensate, in some degree, for the

influence of inter-facility differences on average times

in training, the training measures for the evaluation

samples were standardized relative to the historical

means and standard deviations for each center to

which subjects were assigned. The analysis of variance was then performed on these standardized training progress measures. The working hypothesis for

analysis of the training measures was:

H2

: The progress of MnATCTC graduates through

field training will be the same as FAA Academy graduates, as indexed by standardized days spent, hours of

OJT taken, and indicators of performance earned in

each phase of en route field training.

Attrition

Continued employment in the ATCS occupation,

according to FAA Order 3330.30C (FAA, 1984), is

contingent upon satisfactory progression to the full

performance level. Failure to progress in training may

be the basis for separation from the GS-2152 occupation. Alternatively, an individual not progressing

satisfactorily in field training, as described under

FAA Order 3120.24A (FAA, 1993), might be retained in the ATCS occupation, if, and only if, he or

she has “shown potential for work at the full performance level in different facilities” (FAA, 1984, p. 3).

In other words, two mutually exclusive attrition

outcomes are possible at the first assigned facility.

The trainee might be (a) separated from the ATCS

occupation, or (b) offered the option to switch to a

less demanding facility, such as a lower-level terminal

facility or a Flight Service Station (FSS). Attrition, in

either form, represents a significant economic cost to

the agency. Attrition from ATCS training has been

and continues to be a significant concern to the U.S.

Congress as well (U.S. Congress, 1976; General Accounting Office, 1989b). Attrition rates for the two

evaluation groups were compared, using Fisher’s Ztest under the following hypothesis:

H3

: MnATCTC and FAA Academy graduate attrition rates, defined as separations from the occupation

and switches from the first assigned en route facility, will

be equal.

Performance ratings

First, the overall internal consistency of the ratings

provided by instructors, incumbents, and supervisors

was estimated for each major domain of the rating

instrument as a measure of instrument internal reliability. Second, scale scores were computed by averaging valid responses across items comprising the

scale. Third, scale scores were correlated by rating

source to assess inter-rater reliability. Finally, domain scores for each group were compared by oneway analysis of variance to test the following

hypothesis:

H4

: The mean ratings of technical skill, technical

knowledge, teamwork, and overall potential to succeed

in the occupation given by supervisors and OJT instructors of MnATCTC and FAA Academy graduates, and by

the graduates themselves, will be equal.

As discussed previously, institutions participating

in the CTI/ATCS program had raised concerns about

how two factors might influence the subjective
14

performance ratings. The first factor was acceptance

in the facility, assessed by the incumbent’s and other’s

perceptions. The second factor was the delay between

graduation and hiring for MnATCTC graduates.

Therefore, three additional analyses of variance were

conducted, controlling separately and jointly for (a)

the degree to which the incumbent felt accepted in

the facility, (b) the degree to which the supervisor

perceived the controller as being accepted in the

facility, and (c) hiring delay.

Costs and benefits

As no formal statistical analyses are associated with

cost-benefit comparisons, the research issues are

framed as questions, rather than formal hypotheses.

The first cost-benefit analysis focused on a comparison of the cost-per-hire between the MnATCTC and

the FAA Academy programs. The research question

addressed in this first analysis was:

Q1

: At what point might the MnATCTC cost-perhire be equal to or less than the cost-per-controller at the

FAA Academy?

The second analysis evaluated the ratio of accrued

and projected costs, and benefits for the MnATCTC

program to address the research question:

Q2

: At what point are the benefits accruing from the

MnATCTC program likely to balance or exceed the

direct and indirect costs of the program, e.g., return one

dollar or more in savings to the FAA for each dollar

invested?

RESULTS

Employee diversity

There were no significant differences by participating institution in the representation of minorities

(Table 4). However, there were significantly more

women in the MnATCTC sample (39.7%) than in

the FAA Academy sample (17.2%; Z = 3.67, p <

.001). As a result, the proportion of male non-minorities in the MnATCTC sample was significantly

smaller (53.7%) than in the Academy graduates

(67.2%; Z = -2.55, p < .01), apparently due to the

larger proportion of non-minority females in the

MnATCTC sample.

Field training progress

Descriptive statistics for the days, hours on-thejob training, and indicator of performance (IP) rating

for each phase of en route training by program are

presented in Table 10. Fewer days in phase and fewer

hours of OJT represent better performance; conversely, a higher IP indicates better performance. On

one hand, the one-way analyses of variance of raw

training data by group indicated that MnATCTC

graduates did significantly better than the comparison group of FAA Academy graduates on the following training measures: number of days and hours of

OJT in PHASE VI; hours of OJT in PHASE VIII; days

in PHASE IX, with and without adjustment for the

number of sectors on which training was provided;

days and hours of OJT in PHASE XI; and days in PHASE

XIII. On the other hand, Academy graduates appeared to do better than MnATCTC graduates on

the following training measures: PHASE VII days and

hours of OJT; hours of OJT in PHASE IX, with and

without adjusting for the number of sectors on which

training was provided; and hours of OJT, adjusted

for number of training sectors, in PHASE X and PHASE XI.

Analyses based on raw data, without consideration

of inter-facility differences, are misleading, however.

As noted above, previous research has found substantive differences on training progress measures between en route facilities not attributable to individual

differences in the abilities of trainee controllers (Office of the Deputy Associate Administrator for Appraisal, 1989). A more realistic appraisal of the progress

of MnATCTC and FAA Academy graduates is provided by an analysis of training measures standardized with respect to facility historical means and

standard deviations for each phase. For example, the

days in PHASE IX for a MnATCTC or FAA Academy

graduate assigned to Minneapolis center would be

standardized with respect to the historical mean and

standard deviation of days in PHASE IX at that facility.

The results of the comparison of MnATCTC to

FAA Academy graduates, using standardized training

measures, are presented in Table 11. A negative

standardized score for the days and hours OJT in

phase indicate better performance than average. That

is, a negative score indicates that a person took fewer

days or hours of OJT than average. A positive standardized score for IP indicates a rating higher than

average.

On one hand, graduates from the MnATCTC

program performed statistically better than FAA Academy graduates on six standardized training measures.

The graduates from the MnATCTC program required statistically fewer standardized days (0.02

standard deviations above facility average) in the
15

Table 10. En route days in phase, hours OJT in phase, and phase Indicator of Performance for MnATCTC and Academygraduates

MnATCTC (N=136) FAA Academy (N=157)

Phase Description Measure M SD N M SD N F

V Assistant Controller Days in Phase 40.21 7.42 136 41.47 29.87 157 0.23

Hours OJT 227.47 40.75 136 225.20 54.89 157 0.16IP in Phase 4.08 0.85 106 4.24 0.77 148 2.69

VI Assistant Controller Qual Days in Phase 15.68 13.15 134 23.30 25.76 155 9.58**

Hours OJT 41.43 36.41 134 49.72 25.31 155 5.17*IP in Phase 4.11 0.89 100 4.27 0.60 139 2.65

VII Prelim Nonradar/Radar Days in Phase 54.44 36.75 133 45.31 29.58 148 5.31*

Hours OJT 247.88 82.22 133 212.97 108.02 148 9.13**IP in Phase 4.00 0.88 102 4.13 0.71 143 1.70

VIII Initial Radar Associate Qual Days in Phase 91.53 45.23 133 82.79 31.45 151 3.64

Hours OJT 351.62 212.90 133 403.23 165.72 151 5.26*IP in Phase 3.66 1.18 103 3.90 0.95 147 3.31

IX Initial Radar Associate Qual Days in Phase 62.27 54.55 22 113.64 87.65 91 6.89**

Hours OJT 210.70 62.28 123 170.33 57.52 144 30.27***IP in Phase 3.86 1.18 96 4.25 0.71 127 9.20**Sectors 2.00 121 2.00 139Adj Daysa 31.14 27.27 22 56.91 44.07 90 6.85**

Adj Hoursa 105.25 31.32 121 85.05 28.72 139 29.42***

(Table 10 continues)
16

(Table 10 continued)

MnATCTC (N=136) FAA Academy (N=157)

Phase Description Measure M SD N M SD N F

X Final Radar Associate

Qual

Days in Phase 149.96 78.06 85 166.62 135.56 117 1.10

Hours OJT 202.93 80.49 85 194.44 69.09 117 0.65IP in Phase 4.23 0.81 71 4.12 0.65 107 0.88Sectors 5.09 1.17 80 6.13 1.33 106 31.08***Adj Daysa 28.80 12.00 80 32.31 49.76 106 0.38

Adj Hoursa 40.95 16.24 80 32.87 13.21 106 14.00***

XI Radar Controller

Training

Days in Phase 56.21 22.10 76 72.53 25.80 139 21.68***Hours OJT 267.17 165.97 76 351.17 120.32 139 18.17***IP in Phase 4.11 1.11 64 4.02 1.01 131 0.29

XII Initial Radar Position

Qual

Days in Phase 155.45 67.87 47 142.11 78.38 133 1.08Hours OJT 232.26 72.30 47 203.30 64.47 133 6.57IP in Phase 4.02 0.99 41 4.17 0.77 124 0.94Sectors 2.00 46 2.00 132Adj Daysa 77.49 34.27 46 69.53 35.15 132 1.77Adj Hoursa 115.77 36.47 46 101.57 32.34 132 6.15*

XIII Final Radar Position

Qual

Days in Phase 143.48 67.56 23 202.60 137.85 109 4.01*Hours OJT 199.65 86.22 23 248.77 119.31 110 3.51IP in Phase 4.48 0.68 21 4.30 0.65 104 1.29Sectors 2.61 0.66 23 4.01 09.1 105 48.36***Adj Daysa 56.63 26.22 23 53.08 36.90 105 0.19

Adj Hoursa 78.49 31.85 23 65.42 31.65 107 0.00

Notes: aAdjusted for number of sectors trained on (Days or Hours divided by sectors) *p < .05, **p < .01, ***p < .001
17

Preliminary Nonradar/Radar Associate phase of field

training (PHASE VII) than FAA Academy graduates

(0.14 standard deviations more than facility averages;

F(1, 279) = 5.68, p < .05). MnATCTC graduates

required statistically fewer standardized days of training in PHASE IX (-0.85 standard deviation units fewer

than average) than FAA Academy graduates (-0.13

standard deviation units fewer than average; F(1,110)

= 10.34, p < .01). The same pattern held when the

days in PHASE IX were adjusted for the number of

sectors on which controllers trained (MnATCTC = -

0.84 standard deviations fewer than average, versus -

0.13 standard deviations fewer than average for

Academy graduates; F(1,110) = 10.15, p < .01).

MnATCTC graduates also took fewer standardized

days (0.02 standard deviations more than facility

averages) in Final Radar Association Qualification

training (PHASE X) than FAA Academy graduates

(0.50 standard deviations more than facility averages;

F(1, 200) = 8.60, p < .01). Adjusting the days in

PHASE X training for the number of sectors on which

controllers were trained did not change this pattern

of results; MnATCTC required fewer adjusted days

in training (just 0.11 standard deviations more than

facility averages) than FAA Academy graduates (0.58

standard deviations more than facility averages;

F(1,184) = 5.84, p < .05). The MnATCTC graduates

also required statistically fewer standardized days in

Radar Controller Training (PHASE XI) than FAA

Academy graduates. MnATCTC graduates required

0.51 standard deviations more than facility averages

to complete Phase XI, while FAA Academy graduates

required 0.71 standard deviations more than facility

averages in the same training (F(1,213) = 5.18, p < .05).

On the other hand, FAA Academy graduates performed better than MnATCTC graduates on three

standardized training measures. FAA Academy graduates earned higher standardized IPs, relative to facility averages, in PHASE VI (0.33 standard deviations

above facility averages) than did MnATCTC graduates (just 0.03 standard deviations above facility

averages; F(1,237) = 5.05, p < .05). The FAA Academy graduates took fewer standardized days (0.24

standard deviations more than facility averages) in

the Initial Radar Association Qualification training

phase (PHASE VIII) than MnATCTC graduates (0.50

standard deviations more than facility averages; F(1,

282) = 5.22, p < .05). The FAA Academy graduates

required statistically fewer standardized hours of OJT

in PHASE X (0.44 standard deviations more than

facility averages), adjusted for the number of sectors

on which they trained, than did MnATCTC graduates (0.71 standard deviations above facility averages;

F(1, 184) = 4.29, p < .05).

Training outcomes

Attrition

Outcomes at the first assigned field facility by

school are presented in Table 12. Available information indicated that 10 (7.4%) MnATCTC graduates

had attrited from the FG-2152 occupation as of June

1995. Just 6 (3.8%) FAA Academy graduates had

attrited from their first assigned facility as of June

1995. The attrition rates were not significantly different (Z = 0.91, ns). Significantly more FAA Academy graduates switched options or facilities (N = 7;

4.5%), prior to reaching the full performance level

(FPL) at their first assigned facility, than did

MnATCTC graduates (N = 1, or 0.7%; Z = 1.99,

p < .05). Statistically more MnATCTC graduates

(75.0%) were still in training at their first assigned

facility than FAA Academy graduates (22.9%; Z =

8.91, p < .001). Overall, significantly fewer

MnATCTC had achieved FPL as of June 1995 (N =

23, or 16.9%) than FAA Academy graduates (N =

108, or 68.8%; Z = -8.91, p < .001).

Survival analysis

Survival analysis was also used to examine the

proportions of FAA Academy and MnATCTC graduates remaining in the first assigned facility, that is,

who had neither been separated from the occupation

nor had switched facilities or options as of June 1995.

Survival analysis is a useful technique for examining

the interval between two events, such as enrollment

in field training and attrition, when the second event

(attrition) does not necessarily happen to everyone,

and when subjects are observed for different periods

of time (Norusis, 1990), as is the case in this evaluation. In survival analysis, the overall period of observation is subdivided into intervals. For this summative

evaluation of the MnATCTC program, the period

between enrollment on the job for each subject and

June 30, 1995, or attrition from the first facility, was

the observation period. The observation period was

subdivided into one month intervals using the SPSS

SURVIVAL procedure. For each one month interval,

all subjects who were observed for at least that long

were used to calculate the probability of attrition

occurring in that interval by the SURVIVAL
18

Table 11. Standardized en route days in phase, hours OJT in phase, and phase Indicator of Performance forMnATCTC and Academy graduatesa

MnATCTC FAA Academy

Phase Description Measure M SD N M SD N F

V Assistant Controller Zdays Phase 0.31 0.57 136 0.34 1.51 157 0.05

ZHours OJT 0.55 0.55 136 0.46 0.74 157 1.58ZIP Phase 0.02 0.90 106 0.01 0.88 148 0.02

VI Assistant Controller Qual ZDays Phase 0.03 0.96 134 -0.08 1.02 155 0.23

ZHours OJT -0.27 1.25 134 -0.23 0.74 155 0.10ZIP Phase 0.03 1.17 100 0.33 0.87 139 5.05*

VII Prelim Nonradar/Radar ZDays Phase 0.02 0.51 133 0.14 0.55 148 5.68*

ZHours OJT 0.51 0.71 133 0.43 0.89 148 3.45ZIP Phase -0.09 0.96 102 -0.09 0.85 143 0.01

VIII Initial Radar Associate

Qual

ZDays Phase 0.50 0.74 133 0.24 0.74 151 5.22*ZHours OJT 0.26 1.08 133 0.21 0.79 151 0.24ZIP Phase 0.02 0.98 103 0.16 0.85 147 1.46

IX Initial Radar Associate

Qual

ZDays Phase -0.85 0.48 22 -0.13 1.02 91 10.34**ZHours OJT 0.64 0.80 123 0.51 0.86 144 1.71ZIP Phase 0.08 1.12 96 0.27 0.68 127 2.42Sectors 2.00 121 2.00 139Adj ZDaysa -0.84 0.48 22 -0.13 1.03 90 10.15**

Adj ZHoursa 0.65 0.81 121 0.52 0.88 139 1.40

(Table 11 continues)
19

(Table 11 continued)

MnATCTC FAA Academy

Phase Description Measure M SD N M SD N F

X Final Radar Associate

Qual

ZDays Phase 0.02 78.06 1.12 0.50 1.17 117 8.60**

ZHours OJT 0.47 0.90 85 0.49 0.75 117 0.40ZIP Phase 0.21 0.88 71 0.07 0.69 107 1.50Sectors 5.09 1.17 80 6.13 1.33 106Adj ZDaysa 0.11 0.80 80 0.58 1.58 106 5.84*

Adj ZHoursa 0.70 0.94 80 0.44 0.77 106 4.29*

XI Radar Controller Training Zdays Phase 0.51 0.76 76 0.71 0.52 139 5.18*

ZHours OJT 1.14 0.99 76 1.01 0.64 139 1.29ZIP Phase 0.03 1.07 64 0.03 0.99 131 0.00

XII Initial Radar Position Qual ZDays Phase 0.64 1.05 47 0.55 1.27 133 0.18

ZHours OJT 0.71 0.79 47 0.56 0.91 133 0.95ZIP Phase 0.04 0.93 41 0.23 0.67 124 2.00Sectors 2.00 46 2.00 132Adj ZDaysa 0.66 1.09 46 0.54 1.21 132 0.35

Adj ZHoursa 0.71 0.80 46 0.57 0.91 132 0.95

XIII Final Radar Position Qual Zdays Phase -0.19 0.69 23 0.16 0.97 109 2.71

ZHours OJT 0.48 1.01 23 0.44 0.96 110 0.05ZIP Phase 0.25 0.65 21 0.14 0.63 104 0.58Sectors 2.61 0.66 23 4.01 0.91 105Adj ZDaysa 0.66 0.93 23 0.59 1.00 105 0.09

Adj ZHoursa 1.21 0.96 23 0.84 0.90 107 3.18

Notes: aAdjusted for number of sectors trained on (Days or Hours divided by sectors) *p < .05, **p < .01, ***p < .001
20

procedure. The result is an estimate, for each group,

of the cumulative proportion of graduates remaining

at the first assigned facility at one, two, three and so

on months after enrollment in field training. The

SPSS SURVIVAL procedure uses the Wilcoxon or

Gehan statistic to test the hypothesis that the survival

distributions are the same for MnATCTC and FAA

Academy samples (Norusis, 1990, p. 244).

The results of the survival analysis are presented in

Figure 1. More than 90% of both FAA Academy and

MnATCTC graduates had survived at the first assigned facility, as would be expected from the simple

rate of attrition analysis reported above. The survival

distributions were not statistically different (Wilcoxon

= 2.987, df = 1, p = .084, ns) for the two groups. This

analysis suggested that, after taking into account the

Table 12. Outcomes at first assigned facility

Outcome MnATCTC

FAA

Academy Total

Attrited from 2152 10

(7.4%)

6

(3.8%)

16

(5.5%)

Moved from 1st fac 1

(0.7%)

7

(4.5%)

8

(2.7%)

Still developmental 102

(75.0%)

36

(22.9%)

138

(47.1%)

Made FPL 23

(16.9%)

108

(68.8%)

131

(44.7%)

Total 136 157 293

Figure 1. MnATCTC and FAA Academy attrition/survival analysis
21

different lengths of time the groups had been at their

first assigned facilities, there were no statistically

significant differences in the proportions of

MnATCTC and FAA Academy graduates that would

be expected to remain (e.g., neither be separated from

the occupation nor switch option or facility) at the

first assigned facility over time.

FPL Certification

Time to FPL

Just 23 (16.9%) MnATCTC graduates were certified as FPLs at their first assigned field facility as of

June 1995. In contrast, 68.8% of the FAA Academy

graduates had attained FPL certification by June

1995. This is consistent with the greater time at the

first assigned facility for FAA Academy graduates:

with longer times in the facility, a larger proportion

of FAA Academy graduates would be expected to

have completed the field training sequence. The

average number of years to certification, as shown in

Table 13, was not significantly different between the

two programs. MnATCTC graduates required about

2.82 (SD = 0.59) years to certify, in comparison to

3.18 (SD = 0.53) years for FAA Academy graduates

(F(1,130) = 0.00, ns). Times to FPL were also standardized, with respect to historical facility averages,

and compared. Both groups required slightly more

time to FPL than historical averages (0.34 standard

deviations more than average for MnATCTC graduates, compared with 0.55 standard deviations more

than average for FAA Academy graduates). However,

the standardized times to FPL were not statistically

different for the two groups (F(1, 130) = 0.22, ns).

Accession to FPL

Survival analysis was used to evaluate, from a

different perspective, the proportion of surviving

graduates by program and option still in training

(developmental status), as of June 1995, relative to

their enrollment date. The terminating event in this

analysis was making FPL. The analysis provides comparative data about when program graduates made

FPL, taking into account the different lengths of

observation. The results of this second survival analysis for the Academy and MnATCTC graduates who

did not attrite in the enroute option are presented in

Figure 2. Taking into account the differing amounts

of time in the field, the survival distributions for the

two groups are not statistically different (Wilcoxon =

0.039, df = 1, ns). This analysis suggests that, with

time, MnATCTC graduates can be expected to achieve

FPL certification at about the same time and at about

the same rates as FAA Academy graduates.

Performance ratings

Reliability

Scale scores for 4 domains were computed from

the returned surveys: TEAMWORK (15 items); degree of

acceptance (ACCEPTANCE) in the facility (5 items);

TECHNICAL SKILL (11 items); and TECHNICAL KNOWLEDGE (11 items). Scale scores were computed as the

average of valid responses to the items comprising a

scale. A scale score of 1 indicated a low degree, 2 an

acceptable degree, 3 a higher degree, and 4 the highest

degree of ACCEPTANCE, TEAMWORK, TECHNICAL SKILL,

or TECHNICAL KNOWLEDGE. Estimates of Cronbach’s

alpha (α), a measure of internal consistency, for the

Table 13. Years and standardized years to full performance level (FPL) in the en route

option

MnATCTC FAA Academy

M SD N M SD N F

Years to FPL 2.82 0.59 23 3.18 0.53 108 0.00

ZYears to FPLa 0.34 0.87 23 0.55 0.70 108 0.22

Notes: a

Years to FPL standardized with respect to historical means and standard deviations for facility

type and level to which person was assigned
22

Figure 2. MnATCTC and FAA Academy accession to FPL over

time analysis

TEAMWORK scale were .94 for OJT instructors, .94

for controllers, and .96 for supervisors. Internal

consistency estimates for the 5-item ACCEPTANCE

scale were lower and barely acceptable, ranging

from .42 for supervisors to .59 for controllers. The

TECHNICAL SKILL and TECHNICAL KNOWLEDGE scale

internal consistency estimates were above .90 for

all three rating sources.

The degree of inter-rater reliability on the scales

was assessed by computing correlations between instructors, controllers, and supervisors. Inter-rater

correlations of instructors with controller and supervisor ratings were generally low for TEAMWORK, TECHNICAL SKILL, TECHNICAL KNOWLEDGE, and ratings of

potential to succeed (POTENTIAL), as shown in Table

14. Correlations of controller with supervisor ratings

were moderate for TEAMWORK, TECHNICAL SKILL, and

TECHNICAL KNOWLEDGE. The correlation between controller self- and supervisory evaluation of POTENTIAL

to succeed in the occupation was good (r = .72, p <

.001). Overall, the pattern of results indicated a

relatively low degree of agreement between raters by

source. Therefore, the ratings data were analyzed by

rating source.

OJT Instructor

The analysis of OJT instructor ratings by performance domain by school is presented in Table 15.

Instructor ratings were returned for 53 MnATCTC

and 61 FAA Academy graduates. The instructor

evaluation of the degree to which the person rated was

accepted (ACCEPTANCE) in the facility did not differ by

school (F(1,113) = 1.06, ns). That is, OJT instructors

perceived MnATCTC graduates as being as well

accepted in the facility as FAA Academy graduates.

The mean instructor ratings on TEAMWORK for

MnATCTC and FAA Academy graduates were not

statistically different (F(1,113) = 0.65, ns). In other

words, OJT instructors perceived the teamwork of

MnATCTC graduates as being about the same as that
23

Table 14. Performance ratings dimension inter-rater correlations by rating source

Rating Dimension

Teamwork Accepted Technical Skill Tech Knowledge Success Potential

Dimension Rater M SD N OJTI ATCSSUPV OJTI ATCSSUPV OJTI ATCSSUPV OJTI ATCSSUPV OJTI ATCSSUPVTEAMWORK OJT-Inst 2.35 0.74 114 94

Controller 2.56 0.64 229 39** 96

Supervisor 2.33 0.67 159 35** 65** 96

ACCEPTANCE OJT-Inst 2.76 0.64 114 54** 26** 12 58

Controller 2.73 0.64 229 09 45** 44** 15 69

Supervisor 2.78 0.62 158 21* 47** 48** 11 57** 66

SKILL OJT-Inst 2.00 0.67 114 79** 35** 37** 46** 16 18 96

Controller 2.17 0.67 226 37** 71** 51** 27** 29** 28** 41** 97

Supervisor 2.00 0.61 156 43** 49** 79** 26** 27** 31** 46** 64** 97

KNOWLEDGE OJT-Inst 2.44 0.63 114 68** 36** 34** 54** 18* 25* 73** 31** 41** 91

Controller 2.53 0.62 228 29** 64** 41** 18* 33* 37** 36** 70** 44** 33** 95

Supervisor 2.41 0.62 158 23* 46** 69** 06 37** 37** 30** 50** 74** 28** 63** 96

POTENTIAL OJT-Inst 82.93 17.49 112 69** 51** 41** 41** 31** 46** 69** 40** 45** 63** 32** 28**

Controller 87.96 15.92 230 49** 70** 49** 29** 34** 28** 51** 62** 47** 44** 58** 46** 49**Supervisor 83.67 17.05 157 50** 49** 65** 23* 30** 34** 51** 53** 67** 43** 38** 57** 52** 72**

Note: Decimals omitted for correlations. *p < .05, **p < .01, ***p < .001
24

Table 15. OJT Instructor (OJT-I) ratings on performance dimensions

MnATCTC FAA Academy

M SD N M SD N F

TEAMWORK 2.29 0.76 53 2.40 0.72 61 0.65

ACCEPTANCE 2.70 0.65 53 2.81 0.50 61 1.06

TECHNICAL SKILL 1.83 0.65 53 2.16 0.66 61 7.17**

TECHNICAL KNOWLEDGE 2.28 0.66 53 2.57 0.58 61 6.33*

POTENTIAL 79.65 18.74 52 85.77 15.95 60 3.48

*p < .05, **p < .01, ***p < .001

of FAA Academy graduates. However, OJT instructors rated the TECHNICAL SKILL of MnATCTC graduates as being lower than that of FAA Academy

graduates (F(1,113) = 7.17, p < .01). These same

instructors also evaluated the TECHNICAL KNOWLEDGE

of MnATCTC graduates as being lower than that of

FAA Academy graduates (F(1,113) = 6.33, p < .05).

However, the overall POTENTIAL of MnATCTC graduates to succeed in the occupation (M = 79.65, SD =

18.74 on a 40 to 100 scale) was not rated significantly

lower than that of FAA Academy graduates (M =

85.77, SD = 15.95; F(1,111) = 3.48, ns).

Controller

Overall, the mean scale scores for FAA Academy

graduates across all five rating dimensions were statistically higher than the mean scale scores for

MnATCTC controllers when computed on self-ratings (Table 16). The average self-rating on TEAMWORK

for FAA Academy graduates was 2.68 (on a 1 to 5

scale), compared with a mean of 2.42 for MnATCTC

self-evaluations of TEAMWORK (F(1,228) = 9.71, p <

.01). FAA Academy graduates felt themselves to be

better accepted at the facility (ACCEPTANCE: M = 2.85,

SD = 0.56) than did MnATCTC graduates (M =

2.60, SD = 0.69; F(1,228) = 9.18, p < .01). FAA

Academy graduates also rated their TECHNICAL SKILL

more highly (M = 2.33, SD = 0.55) than did

MnATCTC graduates (M = 1.98, SD = 0.75; F(1,225)

= 16.72, p < .001). Similarly, FAA Academy graduates rated their TECHNICAL KNOWLEDGE of air traffic

control more highly (M = 2.69, SD = 0.50 on a 1 to

5 scale) than did MnATCTC graduates (M = 2.36,

SD = 0.70; F(1,227) = 16.48, p < .001). Finally, FAA

Academy graduates rated themselves as having greater

POTENTIAL to succeed in the occupation (M = 91.45,

SD = 11.73 on a 40-100 scale) than did the

MnATCTC graduates (M = 84.22, SD = 18.77;

F(1,229) = 12.44, p < .001).

Table 16. Controller self-ratings on performance dimensions

MnATCTC FAA Academy

M SD N M SD N F

TEAMWORK 2.42 0.75 109 2.68 0.51 120 9.71**

ACCEPTANCE 2.60 0.69 109 2.85 0.56 120 9.18**

TECHNICAL SKILL 1.98 0.75 106 2.33 0.55 120 16.72***

TECHNICAL KNOWLEDGE 2.36 0.70 108 2.69 0.50 120 16.48***

POTENTIAL 84.22 18.77 111 91.45 11.73 119 12.44***

*p < .05, **p < .01, ***p < .001
25

Supervisor

The supervisor ratings (Table 17) followed a similar pattern as the controller self-ratings: the mean

ratings assigned by supervisors for FAA Academy

graduates were statistically greater than the mean

ratings for MnATCTC graduates across all five rating

dimensions. The mean scale score of supervisory

ratings on TEAMWORK for FAA Academy graduates (M

= 2.48, SD = 0.62 on a 1 to 5 scale) were statistically

greater than those for MnATCTC graduates (M =

2.11, SD = 0.69; F(1,158) = 12.93, p < .001). Supervisors perceived FAA Academy graduates as being

better accepted at the facility (ACCEPTANCE: M = 2.88,

SD = 0.62) than MnATCTC graduates (M = 2.65, SD

= 0.61, F(1,157) = 5.22, p < .05). The TECHNICAL

SKILL of FAA Academy graduates (M = 2.09, SD =

0.58) was rated more highly by supervisors than was

that of MnATCTC graduates (M = 1.86, SD = 0.63;

F(1,155) = 5.19, p < .05). The mean TECHNICAL

KNOWLEDGE ratings by supervisors followed the same

pattern, with FAA Academy graduates being rated

more highly, on average (M = 2.56, SD = 0.55) than

MnATCTC graduates (M = 2.20, SD = 0.67; F(1,157)

= 13.16, p < .001). Finally, supervisors saw more

POTENTIAL to succeed in the ATCS occupation, on

average, in FAA Academy graduates (M = 86.08, SD

= 15.65, on a 40-100 scale) than in MnATCTC graduates (M = 80.17, SD = 18.47; F(1,156) = 4.65, p < .05).

Influence of degree of acceptance and delay on

supervisor ratings

As noted previously, institutions participating in

the CTI/ATCS program had raised concerns about

the potential influence of the degree to which a

program graduate was accepted at the local facility

on supervisory ratings of skill, knowledge, and

performance. Concerns about the impact of substantial hiring delays on those ratings, due to skill and

knowledge decay over time, were also expressed by

institutional representatives. Therefore, supplemental analyses of covariance were conducted, in which

supervisor ratings were analyzed after taking into

account the effects of acceptance and hiring delay as

covariates. For these analyses, the degree of ACCEPTANCE in the facility was taken from the controller’s

perspective. Procedurally, the controller’s perceived

degree of ACCEPTANCE in the facility and the time

between graduation and hiring were processed as

covariates before the main effect of program

(MnATCTC versus FAA Academy) was analyzed

using the SPSS ANOVA command. Instructor evaluations were not addressed in this analysis, as the

concern expressed by institutional representatives

focused on supervisors.

Degree of acceptance. The results of the analysis

of supervisor ratings across performance dimensions,

controlling for the incumbent specialist’s perceived

degree of ACCEPTANCE in the facility, are presented in

Table 18. The first dimension considered was TEAMWORK. The total variability in the TEAMWORK rating is

subdivided in the analysis into four components in

three steps. First, the amount of the variability attributable to controllers’ ratings of ACCEPTANCE in the

facility was computed:

Total variability = variability due to

ACCEPTANCE + remaining variability.

A test was computed to determine if the amount of

variability explained by degree of ACCEPTANCE (the

covariate) was statistically significant. Second, the

Table 17. Supervisor ratings on performance dimensions

MnATCTC FAA Academy

M SD N M SD N F

TEAMWORK 2.11 0.69 64 2.48 0.62 95 12.93***

ACCEPTANCE 2.65 0.61 63 2.88 0.62 95 5.22*

TECHNICAL SKILL 1.86 0.63 61 2.09 0.58 95 5.19*

TECHNICAL KNOWLEDGE 2.20 0.67 63 2.56 0.55 95 13.16***

POTENTIAL 80.17 18.47 64 86.08 15.65 93 4.65*

*p < .05, **p < .01, ***p < .001
26

Table 18. Supervisor ratings on performance dimensions, controlling for controller’s perceived degree ofACCEPTANCE in the facility

Program

FAA

Dimension MnATCTC Academy Element Label Sum of Squares df F

TEAMWORK 2.08 2.48 Covariate ACCEPTANCE 14.47 1 43.11***

Main effect PROGRAM 2.98 1 8.86**

Model 17.49 2 25.99***Residual 52.03 155Total 69.48 157

TECHNICAL

SKILL

1.84 2.09 Covariate ACCEPTANCE 2.70 1 7.85**

Main effect PROGRAM 1.30 1 3.79Model 4.00 2 5.82**Residual 52.26 152Total 52.56 154

KNOWLEDGE 2.19 2.56 Covariate ACCEPTANCE 7.63 1 23.28***

Main effect PROGRAM 2.90 1 8.84**Model 10.53 2 16.06***Residual 50.49 154Total 61.02 156

Potential 79.94 86.08 Covariate ACCEPTANCE 3811.96 1 14.31***

Main effect PROGRAM 661.63 1 2.48

Model 4473.59 2 8.40***Residual 40748.00 153Total 45221.60 155

*p < .05, **p < .01, ***p < .001
27

amount of remaining variability explained by or

attributable to the differences between programs, as

the main effect, was computed:

Total variability = variability due to ACCEPTANCE +

variability due to PROGRAM + leftover variability.

A test is computed to determine if the amount of

variability explained by PROGRAM, as the main effect,

was statistically significant. Finally, the amount of

variability in TEAMWORK ratings attributable to the

joint effects of ACCEPTANCE and PROGRAM, as the

overall model for the relationships between TEAMWORK

ratings, ACCEPTANCE, and PROGRAM, was computed:

Total variability = variability due to ACCEPTANCE +

variability due to PROGRAM + variability

due to both + residual variability.

A final test was computed to determine if the

amount of variability explained jointly by acceptance

and program was statistically significant.

The mean supervisor’s rating of TEAMWORK for

MnATCTC graduates was 2.08, compared with 2.48

for FAA Academy graduates. The covariate in the

analysis was each controller’s scale score for the

degree to which he or she felt accepted at the facility

(ACCEPTANCE). The covariate accounted for a statistically significant portion of the overall variability in

the supervisory ratings of TEAMWORK (F = 43.11, p <

.001). PROGRAM also accounted for a statistically

significant portion of the overall variability in TEAMWORK supervisory ratings (F = 8.86, p < .01). The

degree of ACCEPTANCE and PROGRAM also jointly accounted for a statistically significant portion of variance in supervisory ratings of TEAMWORK (F = 25.99,

p < .001). This pattern of results suggests that, even

after accounting for the effects of how well the

controller felt accepted at the facility, the MnATCTC

graduates were still rated lower on teamwork than

FAA Academy graduates.

Analysis of the supervisor’s TECHNICAL SKILL ratings presents a different picture. The mean rating for

MnATCTC graduates was 1.84, compared with 2.09

for FAA Academy graduates on the TECHNICAL SKILL

dimension. The degree to which the incumbent controller felt accepted at the facility (ACCEPTANCE) accounted for a statistically significant portion of the

overall variability in supervisory ratings of TECHNICAL

SKILL (F = 7.85, p < .01). However, PROGRAM

(MnATCTC or FAA Academy) did not significantly

affect supervisory ratings of TECHNICAL SKILL. This

analysis suggests that differences in the mean ratings

of the technical skill of MnATCTC and FAA Academy graduates may be attributable to differences in

the degree to which controllers were accepted at the

facility rather than to where they were initially trained.

Analysis of the supervisory ratings of TECHNICAL

KNOWLEDGE present a pattern similar to that of the

TEAMWORK ratings. The mean rating of MnATCTC

graduate TECHNICAL KNOWLEDGE by supervisors was

2.19, compared with 2.56 for FAA Academy graduates on a 1 (low) to 5 (high) point scale. The degree

of ACCEPTANCE, from the controller’s perspective,

accounted for a statistically significant portion on the

variance in supervisory ratings of TECHNICAL KNOWLEDGE (F = 23.28, p < .001). However, even after

accounting for the effects of ACCEPTANCE, the program at which the incumbent was initially trained

(PROGRAM) still accounted for a statistically significant portion of the variance in supervisory ratings of

TECHNICAL KNOWLEDGE (F = 8.84, p < .01). This

pattern suggests that differences in ratings of TECHNICAL KNOWLEDGE for MnATCTC and FAA Academy

graduates cannot be explained away merely as a

consequence of MnATCTC graduates being less well

accepted at the facility. Rather, the difference in supervisory ratings may reflect real differences in the air traffic

control technical knowledge of the two groups.

The pattern of results for the analysis of supervisory ratings of incumbent POTENTIAL to succeed is

similar to that of ratings of TECHNICAL SKILL. The

mean rating of MnATCTC graduate POTENTIAL to

succeed was 79.94, compared with 86.08 for FAA

Academy graduates. The degree to which controllers

felt accepted at the facility (ACCEPTANCE) accounted

for a statistically significant portion of the overall

variability in ratings of POTENTIAL to succeed (F =

14.31, p < .001). However, PROGRAM (MnATCTC or

FAA Academy) did not account for any significant

variance (F = 2.48, ns). This analysis suggests that

differences in the supervisory ratings of POTENTIAL to

succeed might have been influenced by the degree to

which the new controllers were accepted at the facility, but not by the program from which the controller

graduated. In other words, persons perceiving themselves as less well accepted at the facility were also seen

as having less potential to succeed in the ATCS

occupation by their supervisors, regardless of the

person’s hiring source (FAA Academy or MnATCTC).
28

Hiring delay. The second covariate analyzed was

the delay between graduation and hiring. FAA Academy graduates experienced very little delay between

Academy graduation and starting OJT at their first

facility (M = 0.23 months). MnATCTC graduates, in

contrast, experienced average delays of 6 months

between graduation in Minnesota and starting OJT

(M = 6.02 months, SD = 3.07). It was hypothesized

by CTI/ATCS representatives that the hiring DELAY

would lead to a time-based degradation of skills and

knowledge. As a consequence, MnATCTC graduates

might receive lower ratings than FAA Academy graduates who had not experienced those delays and attendant knowledge and skill losses. As with the degree of

ACCEPTANCE, analysis of covariance was used to explore the degree to which hiring delays influenced

supervisory ratings of TEAMWORK, TECHNICAL SKILL,

TECHNICAL KNOWLEDGE, and POTENTIAL to succeed in

the ATCS occupation.

The results of this analysis are presented in Table

19. Hiring DELAY accounted for a statistically significant portion of the overall variability in supervisory

ratings of TEAMWORK (F = 5.30, p < .05). However,

PROGRAM (MnATCTC or FAA Academy) still accounted for a statistically significant portion of variability in TEAMWORK ratings (F = 9.89, p < .01), even

after accounting for the effects of hiring delays. This

pattern of results suggests that differences in the mean

supervisory ratings of TEAMWORK for MnATCTC and

FAA Academy graduates were not merely the result of

delays in hiring but may have reflected real differences between the groups.

In contrast, hiring DELAY was not a statistically

significant factor in explaining the variability of

supervisor’s ratings of TECHNICAL SKILL (F = 2.00, ns).

Differences in the mean ratings were attributable

only to the PROGRAM in which the controller was

initially trained (F = 4.19, p < .05). This analysis

suggests that hiring delays had no effect on the mean

ratings of TECHNICAL SKILL for the two groups.

Supervisory ratings of TECHNICAL KNOWLEDGE followed the same pattern as TEAMWORK, with both

hiring delay and PROGRAM accounting for statistically

significant portions of the overall variability in TECHNICAL KNOWLEDGE supervisory ratings. This pattern of

results suggested that differences in the mean ratings

of TECHNICAL KNOWLEDGE for MnATCTC and FAA

Academy graduates were not merely the result of

delays in hiring, but may have reflected real differences between the groups.

Finally, supervisory ratings of POTENTIAL to succeed in the occupation for the two groups followed

the same pattern as the TECHNICAL SKILL ratings.

Hiring DELAY was not a significant factor in explaining the variability in ratings of POTENTIAL to succeed

for the two groups (F = 0.02, ns), while PROGRAM was

(F = 15.94, p < .001). In other words, differences in

supervisory ratings of POTENTIAL to succeed in the

ATCS occupation could not be attributed to the

delay in hiring MnATCTC graduates.

Joint effects of acceptance and hiring delay. The

last analysis of covariance considered the joint effects

of the degree to which a controller felt accepted at the

facility (aCCEPTANCE), and the DELAY in hiring experienced, on supervisory ratings of TEAMWORK, TECHNICAL SKILL, TECHNICAL KNOWLEDGE, and POTENTIAL to

succeed in the ATCS occupation. The results of this

covariate analysis for TEAMWORK supervisory ratings

are presented in Table 20. ACCEPTANCE was a significant factor in explaining the overall variability in

supervisor ratings of TEAMWORK (F = 43.04, p < .001).

Hiring DELAY was also a significant factor (F = 4.33,

p < .05), as was the combined effects of the two

covariates (F = 23.69, p < .01). However, even after

accounting for the joint effects of ACCEPTANCE and

hiring DELAY, PROGRAM still had a statistically significant main effect on supervisor ratings of TEAMWORK

(F = 5.27, p < .05). These results suggest that differences in TEAMWORK ratings between MnATCTC and

FAA Academy graduates cannot be explained away as

the effects of not being accepted at the facility and

long delays in hiring MnATCTC graduates but may

have reflected real differences in the teamwork of

MnATCTC compared with FAA Academy graduates.

The analysis of supervisory ratings of TECHNICAL

SKILL, taking into account the joint effects of acceptance and hiring delay, leads to a different result

(Table 21). The acceptance covariate was a significant factor in the overall variability of supervisor

ratings of TECHNICAL SKILL (F = 7.83, p < .01) but not

hiring delay (F = 1.59, ns). After accounting for the

effects of the degree to which the controller felt

accepted at the facility, PROGRAM (MnATCTC or

FAA Academy) accounted for a statistically significant portion of the overall variability in ratings of

TECHNICAL SKILL (F = 4.71, p < .01). These results

suggested that the lower ratings on TECHNICAL SKILL

for the MnATCTC graduates, compared with FAA

Academy graduates, could not be explained away by
29

Table 19. Supervisor ratings on performance dimensions, controlling for DELAY between graduation and

reporting to facility

Program

MnATCTC FAA Academy

Dimension (N = 63) (N = 95) Element Label Sum of Squares df F

TEAMWORK 2.08 2.48 Covariate Hiring DELAY 2.21 1 5.30*

Main effect PROGRAM 4.12 1 9.89**

Model 6.32 2 7.59***Residual 64.95 156Total 71.27 158

TECHNICAL

SKILL

1.84 2.09 Covariate Hiring DELAY 0.72 1 2.00

Main effect PROGRAM 1.51 1 4.19*Model 2.23 2 3.09*Residual 55.22 153Total 57.45 155

KNOWLEDGE 2.19 2.56 Covariate Hiring DELAY 2.09 1 5.80*

Main effect PROGRAM 3.30 1 9.17**Model 5.39 2 7.49***Residual 55.79 155Total 61.18 157

Potential 79.94 86.08 Covariate Hiring DELAY 5.78 1 0.02

Main effect PROGRAM 4253.40 1 15.94***

Model 4259.18 2 7.98***Residual 41091.60 154Total 43350.80 156

*p < .05, **p < .01, ***p < .001
30

Table 20. Supervisor ratings of TEAMWORK, controlling for ATCS’s perceived degree of ACCEPTANCE in the facility

and DELAY between graduation and hiring

Program

MnATCTC FAA

Academy

Dimension (N = 63) (N = 95) Element Label Sum of Squares df F

Teamwork 2.08 2.48 Covariate ACCEPTANCE 14.47 1 43.04***

Covariate Hiring DELAY 1.46 1 4.33*

Combined 15.93 2 23.69***Main effect PROGRAM 1.77 1 5.27*Model 17.70 3 17.55***Residual 51.78 154Total 69.48 157

*p < .05, **p < .01, ***p < .001

Table 21. Supervisor ratings of TECHNICAL SKILL, controlling for ATCS’s perceived degree of ACCEPTANCE in the facility

and DELAY between graduation and hiring

Program

MnATCTC FAA

Academy

Dimension (N = 63) (N = 95) Element Label Sum of Squares df F

TECHNICAL SKILL 1.84 2.09 Covariate ACCEPTANCE 2.70 1 7.83**

Covariate Hiring DELAY 0.55 1 1.59

Combined 3.25 2 4.71**Main effect PROGRAM 0.95 1 2.74Model 4.19 3 4.05**Residual 52.07 151Total 56.26 154

*p < .05, **p < .01, ***p < .001
31

differences in the degree to which MnATCTC and

FAA Academy graduates felt accepted at the facility

but may have reflected real differences in performance.

The analysis of covariance for supervisor ratings of

technical knowledge for the two groups is presented

in Table 22. The pattern of results for technical

knowledge was again similar to that of TEAMWORK,

with PROGRAM accounting for a statistically significant portion of the overall variability in TECHNICAL

KNOWLEDGE ratings, even after accounting for the

separate and joint effects of acceptance and hiring

delay. Finally, the analysis of the POTENTIAL to succeed ratings for the two groups is presented in Table

23. The results for POTENTIAL to succeed followed the

same pattern as the TECHNICAL SKILL ratings, with

ACCEPTANCE accounting for a statistically significant

portion of the variability in the POTENTIAL to succeed

ratings made by supervisors (F = 15.04, p < .001).

Hiring DELAY was not a significant factor in this

analysis, while PROGRAM was. The results suggest that

the lower ratings of POTENTIAL to succeed given by

supervisors to MnATCTC graduates in comparison

to FAA Academy graduates could not be explained as

the consequence of differences in the degree to which

MnATCTC graduates were accepted in the facility.

Rather, the supervisor ratings of POTENTIAL to succeed in the ATCS occupation may have reflected real

differences in performance between the two groups.

Costs and benefits

Costs

Program costs. Accrued direct costs for the

MnATCTC program consisted of a series of Congressional earmarks totaling $10.9M. Indirect costs

included: (a) headquarters human resources management staff time; (b) site visits by FAA personnel; (c)

CAMI evaluation staff time; (d) evaluation contracts;

(e) program steering committee meetings; (f) regional liaison staff time; (g) FAA Academy staff time;

and (h) Air Traffic staff time. The total time and costs

associated with managing the CTI/ATCS program,

as reported via e-mail, were prorated across the five

participating institutions, except where those costs

were explicitly attributed to MnATCTC only. A

breakdown of those annual costs by fiscal year (FY)

are presented in Appendix B; the logic of each annual

costs worksheet is described in Table 24. To estimate

the running cost-per-hire for the MnATCTC program, the cumulative costs each year were amortized

over the cumulative actual and projected number of

hires through that year. The cost per hire in 1991 was

high, at about $281,000, reflecting initial start-up

costs for the program. This cost-per-hire was reduced

in 1992 to about $121,000 per MnATCTC graduate

hired by the FAA. That cost was further reduced in

1993 to approximately $81,000 per hire as more

MnATCTC graduates entered FAA service. With

additional hiring in FY94, the MnATCTC cost-perhire was stable at about $57,000 through FY95. The

FY96 cost was projected to be about $50,000 per

graduate with continued hiring.

Cost comparison. Actual and projected hiring for

MnATCTC program graduates was used to estimate

the likely cost per hire through the year 2003, as

shown in Figure 3. The costs per controller at the

FAA Academy under the redesigned “Train to Succeed” model were estimated by the Air Traffic Training Work Group (ATTWG, 1992) at about $33,000

through the PV phase. Slightly different cost figures

can be obtained from the FAA Academy Tuition

Pricing System (ATPS; FAA Academy, 1994). The

per controller cost for basic en route training was

estimated at about $58,000 in FY94 by ATPS. Finally, more recent figures, as coordinated with the

FAA Academy, the Training Requirements Division

(ATZ-100), and Assistant Director of the Office of

Air Traffic Program Management (ATZ-2) as of

April 1995 (Larry Lackey, personal communication,

May 1995), estimated the Academy cost for initial

resident training at $45,500 in the en route option1

.

The highest and lowest estimates were used to define

a range for estimated Academy per-controller costs

for comparison purposes.

This cost analysis suggests that MnATCTC is

competitive with the FAA Academy, in terms of the

costs to produce each graduate at about $50,000 per

controller. With the projected hiring of 64 graduates

in FY97, and as many as 100 in FY98 and beyond,

MnATCTC per graduate costs may be lower than the

FAA Academy’s lowest estimated per controller cost.

Continued hiring of MnATCTC graduates could

reduce the fully amortized cost per MnATCTC hire

to a level that is competitive with internal FAA

1

The per-student expenditure rate at the FAA Academy includes incremental costs and excludes about $13.7 million in capital

costs for the En Route program (Sweetman, personal communication, January 30, 1997).
32

Table 22. Supervisor ratings of TECHNICAL KNOWLEDGE, controlling for ATCS’s perceived degree of ACCEPTANCE in the

facility and DELAY between graduation and hiring

Program

MnATCTC FAA Academy

Dimension (N = 63) (N = 95) Element Label Sum of

Squares

df F

KNOWLEDGE 2.19 2.56 Covariate ACCEPTANCE 7.63 1 23.23***

Covariate Hiring DELAY 1.46 1 4.43*

Combined 9.09 2 13.83***Main effect PROGRAM 1.66 1 5.07*Model 10.75 3 10.91***Residual 50.27 153Total 61.02 156

*p < .05, **p < .01, ***p < .001

Table 23. Supervisor ratings of potential to succeed in occupation, controlling for ATCS’s perceived degree ofacceptance in the facility and delay between graduation and hiring

Program

MnATCTC FAA Academy

Dimension (N = 63) (N = 95) Element Label Sum of

Squares

df F

POTENTIAL 79.94 86.08 Covariate Acceptance 3811.96 1 15.04***

Covariate Hiring delay 8.09 1 0.03Combined 1910.02 2 7.54***Main effect Program 2874.44 1 11.34***Model 6694.48 3 8.80***Residual 38527.10 152Total 45221.60 155

*p < .05, **p < .01, ***p < .001
33

Academy costs, despite an additional $1,700,000 in

funding in FY97 (P.L. 104-205). Even with continuing subsidies to the Minnesota Air Traffic Control

Training Program of $1.5 million per year, as requested in congressional testimony by the Director of

that program (Pointer, March 1995), the cost per

MnATCTC hire is likely to be very competitive with

FAA Academy per controller costs in the en route

option. Elimination of continued subsidies to the

MnATCTC program in fiscal year 1997 and beyond,

as recommended in the 1993 National Performance

Review (p. 98) would only increase the cost advantage of the MnATCTC program over the FAA Academy in providing initial technical training for the en

route option over the long term.

Benefits

Program benefits

A cost advantage, however, does not necessarily

result in benefits to the taxpayer. Therefore, the next

step was to analyze benefits accruing from the

MnATCTC program relative to costs. Four classes of

program benefits that could be clearly expressed in

dollars were identified: (a) avoided screening costs;

(b) avoided Academy training costs; (c) avoided PV

costs; and (d) savings from reduced time to FPL.

Screening costs refers to the agency costs in administering either the former FAA Academy ATCS

Nonradar Screen program (Broach & Manning, 1994)

through March 1992, or the replacement computerized ATCS Pre-Training Screen (Broach & BrechtClark, 1994) from June 1992 through June 1996.

The costs of the former ATCS Nonradar Screen (FAA

Academy course 50321) were estimated at about

$10,000 per controller in 1991. The replacement

computerized test battery cost about $1,500 per

examinee (Broach & Brecht-Clark, 1994). The FAA

avoided incurring these costs for MnATCTC graduates as they did not go through the Nonradar Screen

or the ATCS Pre-Training Screen.

Avoided Academy training costs refers to costs

incurred by the agency under the redesigned Academy program of about $45,500 per controller in the

en route option. The redesigned FAA Academy program will consist of three phases or modules: academics (Phase I); techniques (Phase II); and skills building

(Phase III; Air Traffic Training Group, 1996). There

were no plans for MnATCTC graduates to attend any

phase of the FAA Academy ATCS training program,

as of 1995, thereby saving the agency approximately

$45,500 per year per graduate hired.

Performance verification (PV) is the Air Traffic

evaluation of the readiness of a trainee to enter the

field for on-the-job training. PV was instituted in

June 1992 as part of the overhaul of the ATCS

curriculum by the ATTWG (1992). The core of PV

requires bringing experienced field controllers in as

evaluators for each trainee. The costs of PV based on

(a) five days salary for a FG-14 FPL controller, (b)

five days per diem in Oklahoma City, and (c) an

average round-trip fare of $250. The rounded sum of

these costs was prorated across four examinees to

arrive at a cost estimate of about $500 per PV examinee. From 1992 through the present, the agency

delegated the PV function to the schools under the

supervision and direction of the Air Traffic Performance Verification Division (ATZ-400). Thus, the

agency has avoided those PV costs between 1992 and

the present. There are no plans for MnATCTC graduates to undergo PV at the FAA Academy, and the FAA

will continue to avoid PV costs for these controllers.

The final category of benefit to the agency is in the

reduction of the time to FPL. The 1991 comprehensive review of ATCS training estimated en route field

training costs over a 36 month average time to FPL at

$131,739 (ATTWG, 1991a). This total cost was

prorated over the three years to provide an estimate of

the annual OJT cost of about $43,000. Reductions in

time to FPL reduce the amount spent on training,

which represents a savings to the agency.

Cost-benefit analysis

The cost-benefit analysis focused on determining

the point at which cumulative program benefits would

balance or exceed cumulative MnATCTC program

costs. Costs were calculated from 1991 through the

year 2003 as the sum, each year, of identified direct

and indirect costs, as shown in each fiscal year’s

worksheet in Appendix B and Table 24. Benefits

accrued from avoided costs were calculated for each

year by multiplying the number of MnATCTC graduates hired by the avoided screening, three phases of

initial qualification training at the FAA Academy,

and PV costs, as shown in the cost-benefit analysis

worksheets in Appendix B. The logic of the costbenefit worksheet is presented in Table 25. The
34

Table 24. Annual FY costs worksheet logic

Cost Category Description N Rate Cost ($)Direct = cost attributableor specific to theMnATCTC program

For example, financialsubsidies provided atthe direction ofCongress

Units (not costs)allocated to MnATCTCprogram

Unit costs, for example,annual appropriation toMnATCTC

Units x rate = Cost

Program = indirect costprorated to MnATCTCprogram

For example, FAA staffcosts, evaluation costs,and travel. Staff costsare based on publishedlocality-based pay ratesfor the grade/step of theFAA staff person.

Prorated units (notcosts) or portion of FTEallocated to MnATCTCprogram

Unit costs, for example,GS-14 annual salary

Units x rate = Cost

Annual Costs = Sum of costs for this

fiscal year

Cumulative costs = This year’s costs + all

previous years’ costsback to 1991

Annual hires = This year’s actual or

projected number ofMnATCTC graduates (tobe) hired by FAA

Cumulative hires = This year’s hires + all

previous years’ hiresback to 1991

Cost per hire = Cumulative costs

divided by cumulativehires
35

Figure 3. MnATCTC projected cost-per-hire
36

Table 25. Cost-benefit worksheet logic

Fiscal Year Annual Number of

MnATCTC Program

Hires

Cumulative Number

of MnATCTC

Program Hires

FAA Screening Cost Cumulative Savings

from Avoided

Screening Costs

Fiscal year (refer toFY worksheet)

Actual number ofhires for FYworksheet

Cumulative number ofhires from FYworksheet

Costs for FAAAcademy NonradarScreen FY91-92($10,000) orATCS/PTS FY93-96($1,500). If $0 in cell,then cost not avoided

Number of annualhires for FY xScreening cost forthat year, summedacross years

(Columns continue across page in worksheet)

FAA Academy

Academics (Phase I)

Training Cost

FAA Academy

Techniques (Phase

II) Training Cost

FAA Academy

Skills-building

(Phase III) Training

Cost

Cumulative Savings

from Avoided FAA

Academy Training

Costs

Starting in FY93,estimated at $15,000per student

Starting in FY93,estimated at $15,000per student. If $0entered, cost notavoided. $7,500indicates 50% of costavoided, by havingMnATCTC enter midcourse in Phase II

Starting in FY93,estimated at $15,500per student; $0indicates cost notavoided by FAA

Number of annualhires for FY x (PhaseI + Phase II + PhaseIII avoided costs),summed across years

(Table 25 continues)
37

(Table 25 continued)(Columns continue across page in worksheet)

ATX-400

Performance

Verification Cost per

Student

Cumulative Savings

from Avoided PV

Costs

Reduction in Time to

FPL (Years)

Average OJT Cost

per Year

Annual Savings from

Reduced Time to FPL

Estimated cost perstudent, as describedin text; if $0, cost notavoided

Annual hires x PV costper student, summedacross years

Difference betweenMnATCTC averagetime to FPL and facilityhistorical averages,computed as astandardized Z-score

x Estimated OJT costas described in text

= Annual savings

Cumulative Savings

from Reduced Time

to FPL

Total Cumulative

Savings (Avoided

Screening, Training,

PV, and Time to FPL

Savings)

Cumulative

MnATCTC program

costs

MnATCTC BenefitCost Ratio Cost-per-hire

Annual savings fromreduced time to FPL,summed across years

sum of cumulativeavoided screeningcosts, cumulativeavoided FAA Academytraining costs, avoidedPV costs, pluscumulative savingsfrom reduced time toFPL

Cumulative costs (todate) for MnATCTCfrom FY worksheet

= cumulative costsdivided by cumulativesavings (benefits)

Cumulative cost-perhire from FYworksheet
38

reduction in time to FPL, while not statistically

significant, was multiplied by the average OJT cost

per year to assess the financial utility of even small

gains in efficiency; that product was multiplied by

the number of graduates per year to provide a rough

estimate of savings attributable to reductions in

time-to-FPL. Avoided costs and OJT savings were

then summed.

The cumulative costs and benefits were then compared, as shown in Figure 4, to identify the probable

time frame in which benefits accruing from the

MnATCTC program might balance or exceed cumulative costs. Without considering other factors, such

as the field training resource implications of the

differing skill levels indicated by the ratings data, it

appears that cumulative benefits to the FAA from the

MnATCTC program might outweigh sunk costs by

about FY1998, based on the projected hiring rates.

The ratio of cumulative benefits to costs is illustrated

in Figure 5. Overall, the MnATCTC program might

return at least one dollar in avoided costs and savings

for every dollar invested by FY1998, based on current

projected hiring rates.

DISCUSSION

Overall, the CTI/ATCS program appears to be

meeting its operational objectives in terms of employee diversity, progress in field training, and controller performance. The MnATCTC program appears

to be providing a greater proportion of women to the

field facilities than has been provided through the

FAA Academy. However, the MnATCTC program

does not appear to be a better source for minority

controllers than existing workforce sources such as

the FAA Academy. There are few significant differences between MnATCTC and FAA Academy graduates in terms of training measures after taking into

account inter-facility differences in programs.

MnATCTC graduates appear to do better than FAA

Academy graduates on some training measures, and

FAA Academy graduates do better on others. The net

effect appears to be that, overall, MnATCTC graduates require about the same amount of days and hours

of OJT as FAA Academy graduates. The attrition

rates for the two groups are similar. A relatively small

number of MnATCTC graduates had been certified

Figure 4. MnATCTC cumulative costs and benefits
39

as FPL controllers as of June 1995, compared with a

larger number of FAA Academy graduates. However,

the times required for the two groups to certify were

about the same. Moreover, the accession analysis

suggested that the number of MnATCTC graduates

that could be reasonably expected to achieve FPL,

and their times to FPL, are likely to be comparable to

those for FAA Academy graduates.

On the other hand, the ratings data suggest some

caution in concluding that the MnATCTC is producing graduates that are strictly comparable to graduates of the FAA Academy — in terms of their

teamwork, technical skill, knowledge, and potential

to succeed. Overall, the mean ratings of MnATCTC

graduate teamwork, technical skill, knowledge, and

potential to succeed were statistically lower than the

mean ratings of the Academy comparison group,

even after taking into account other factors, such as

the degree to which graduates felt accepted at the

facility and hiring delays. Alternative explanations

for these difference might include discomfort with

MnATCTC graduates and rating errors associated

with stringency and leniency. Another explanation

might be that FAA ATCS supervisors are as yet

uncomfortable with controllers entering the work

force through other than the traditional pipelines.

Or, it may be that the differences in ratings reflect real

differences in performance. The development of objective measures of the core technical performance of

controllers, such as envisioned for the Separation and

Control Hiring Assessment (SACHA) procurement

(FAA, 1991) and its successor, the Air Traffic Selection and Training (AT-SAT) program (FAA, 1996),

may provide better assessment tools in future program

evaluations than subjective ratings that may be influenced by stringency, leniency, and other rating errors.

Finally, the cost-benefit data suggest that benefits

may accrue to the agency by using MnATCTC as an

alternative workforce recruiting and training source

for the en route option. It appears from this analysis

that MnATCTC can produce graduates at a cost that

is competitive with the FAA Academy, even with

continuing, congressionally-mandated financial support from the FAA for the Minnesota program. The

cost-benefit analysis for MnATCTC also suggests

that, given current hiring projections, the MnATCTC

will have a positive return-on-investment by about

FY1998. The majority of that benefit accrues in the

form of avoided training costs at the FAA Academy.

However, this analysis does not consider other costs

to the agency. For example, the differences in skill

level for MnATCTC graduates may place greater

burdens over time on facility training resources to

bring those graduates up to a common standard of

Figure 5. MnATCTC cumulative benefit-to-cost ratio
40

performance. A significant effort will be required of

the FAA to recruit minority candidates for the ATCS

occupation in view of the difficulties experienced by

the MnATCTC to recruit a diverse student population. Maintaining the technological currency of the

MnATCTC program on the NAS architecture as it

evolves may impose another cost on the agency that

is not reflected in this cost-benefit analysis. Moreover, improvements in efficiency at the FAA Academy, reducing agency costs, would reduce the apparent

financial benefits of the MnATCTC program.

In conclusion, this first summative evaluation

found, on one hand, that the MnATCTC program,

under the Collegiate Training Initiative for Air Traffic Control Specialists (CTI/ATCS) umbrella, appears to be meeting defined program objectives, in

terms of recruiting women into this traditionally

male occupation. However, MnATCTC has been

less successful than the FAA in recruiting minorities

into the ATCS occupation, despite a substantial

investment in a national recruiting program.

MnATCTC graduate progress through the controller

field training appears to be on a par with that of

persons entering the occupation through the FAA

Academy, based on the objective tracking and attrition data. However, the subjective ratings are less

supportive, with MnATCTC graduates having lower

average ratings of teamwork, technical skill, technical

knowledge, and potential to succeed in the ATCS

occupation than Academy graduates, even after controlling for the fact that MnATCTC graduates felt

less well accepted in the facility and experienced

significant delays between graduation and hiring.

These differences may reflect actual performance

differences, or perhaps, discomfort with persons entering the occupation through other than traditional

routes. Research on supervisory attitudes and expectations of new controllers might provide a basis for

understanding these differences in ratings and for

designing training management interventions and

strategies to mitigate any discomfort and ease the

organizational socialization process for new controllers in field facilities. Differences in personality and

biographical background between persons that enter

through the collegiate and competitive channels might

also be investigated as potential explanations for

differences in performance. Objective measures of

core technical performance, currently under development, may be used in future evaluations to provide a

more definitive comparison of the safety and efficiency

of CTI/ATCS graduates to that of controllers who

entered the occupation through the FAA Academy.

Finally, the cost-benefit analysis suggests some longterm benefit for the FAA by utilizing the MnATCTC

program as an alternative workforce source. Overall,

the pattern of results in this first summative evaluation suggests that expansion of the CTI/ATCS program to additional educational institutions might be

considered by the FAA as part of an overall strategy

for staffing the National Airspace System.

REFERENCES

Air Traffic Control Specialist Hiring & Training

Workgroup (1996). Air traffic control specialist

hiring and training workgroup results and recommendations. Washington, DC: Federal Aviation

Administration Office of Air Traffic Program

Management Career Systems Division (ATZ200).

Air Traffic Training Workgroup (1991a). ATTWG

Training plan: Overview. Washington, DC: Federal Aviation Administration Office of Air Traffic

Program Management Training Requirements

Division (ATZ-100).

Air Traffic Training Workgroup. (1991b). Controller

resource management conference: Summary of the

meeting. Washington, DC: Federal Aviation Administration Office of Air Traffic Program Management Training Requirements Division (ATZ100).

Bloom, B. (1967). Toward a theory of testing which

includes measurement-evaluation-assessment.

(CSEIP Occasional Report No. 9). Los Angeles,

CA: University of California at Los Angeles Center for the Study of Evaluation of Instructional

Programs.

Borman, W.C., White, L.A., Pulakos, E.D., & Oppler,

S.H. (1991). Models of supervisory job performance ratings. Journal of Applied Psychology, 76,

863 – 72.

Broach, D. & Brecht-Clark, J. (1994). Validation of the

Federal Aviation Administration air traffic control

specialist Pre-Training Screen. (DOT/FAA/AM94/4). Washington, DC: Federal Aviation Administration Office of Aviation Medicine. (NTIS

No. ADA277549)
41

Broach, D. & Manning, C.A. (1994). Validity of the air

traffic control specialist Nonradar Screen as a predictor of performance in radar-based air traffic control

training. (DOT/FAA/AM-94/9). Washington,

DC: Federal Aviation Administration Office of

Aviation Medicine. (NTIS No. ADA279745)

Campbell, J.P. (1990). Modeling the performance prediction problem in industrial and organizational

psychology. In M.D. Dunnette & L.M. Hough

(Eds.), Handbook of industrial and organizational

psychology, Volume 2 (2nd Ed.), pp. 687 – 732.

Palo Alto, CA: Consulting Psychologist Press.

Campbell, J. & Cook, D.T. (1976). The design and

conduct of quasi-experiments and true experiments in field settings. In M.D. Dunnette (Ed.),

Handbook of industrial and organizational psychology (pp. 223 – 326). Chicago, IL: Rand McNally.

Campbell, J.P., McCloy, R.A., Oppler, S.H., & Sager,

C.E. (1993). A theory of performance. In N.

Schmitt, W.C. Borman and Associates (Eds.),

Personnel selection in organizations (pp. 35 – 70).

San Francisco: Jossey-Bass.

Department of Transportation. (1993, May). Report on

the audit of Federal Aviation Administration sponsored higher education programs. (DOT/OIG Report AV-FA-010). Washington, DC: Department

of Transportation, Office of the Inspector General.

FAA Academy. (1994, November). Academy Tuition

Pricing System. Oklahoma City, OK: Program

and Project Management Division (AMA-100).

Federal Aviation Administration. (1984). Employment

program for developmental air traffic control specialists. (FAA Order 3330.30C, September 27,

1984). Washington, DC: Author.

Federal Aviation Administration. (1985). National air

traffic training program tracking system. (FAA Order 3120.22A, July 1, 1985). Washington, DC:

Author.

Federal Aviation Administration. (1988). National air

traffic training program en route instructional program guide. (EP 12-0-1 E). Oklahoma City, OK:

Federal Aviation Administration Academy.

Federal Aviation Administration. (1993). Air traffic

control specialist on-the-job training and position

certification. (FAA Order 3120.24A, Change 1,

August 30, 1993). Washington, DC: Author.

General Accounting Office (1989a). FAA training: Continued improvements needed in FAA’s controller

field training program. (GAO/RCED-89-93).

Washington, DC: Author.

General Accounting Office (1989b). Aviation safety:

Serious problems continue to trouble the air traffic

control work force. (GAO/RCED-89-112). Washington, DC: Author.

E.L. Hamm & Associates Inc., & Hampton University.

(1990, July). Preliminary draft of the research plan

for the air traffic control prototype curriculum. (Interim report submitted under FAA contract

DTFA01-90-Y-01034). Washington, DC: Federal Aviation Administration Office of Training

and Higher Education.

Hartel, C.E.J., & Hartel, G.F. (1995). Controller resource management — What can we learn from

aircrews? (DOT/FAA/AM-95/21). Washington,

DC: Federal Aviation Administration Office of

Aviation Medicine. (NTIS No. ADA297386)

Hoffman, C.C., Nathan, B.R., & Holden, L.M. (1991).

A comparison of validation criteria: Objective

versus subjective performance measures and selfversus supervisor ratings. Personnel Psychology, 44,

601 – 18.

Manning, C.A. (1990). Attrition from ATCS field training. [Unpublished manuscript]. Oklahoma City,

OK: Federal Aviation Administration Civil Aeromedical Institute.

Manning, C.A., Della Rocco, P.S., & Bryant, K.D.

(1989). Prediction of success in FAA air traffic

control field training as a function of selection and

screening test performance. (DOT/FAA/AM-89/

6). Washington, DC: Federal Aviation Administration Office of Aviation Medicine. (NTIS No.

ADA209327)

Means, B., Mumaw, R. Roth, C., Schlager, M,

McWilliams, E., Gagne, E., Rice, V., Rosenthal,

D., & Heon, S. (1988). ATC training analysis

study: Design of the next-generation ATC training

system. (Final report delivered under OPM work

order 342-036). Alexandria, VA: HumRRO International, Inc.

Morrison, J.E., Fotohui, C.H., & Broach, D. (1996). A

formative evaluation of the Collegiate Training Initiative - Air Traffic Control Specialist (CTI-ATCS)

program. (DOT/FAA/AM-96/6). Washington,

DC: Federal Aviation Administration Office of

Aviation Medicine. (NTIS No. ADA305307)
42

National Performance Review. (1993, September). From

red tape to results: Creating a government that works

better and costs less. Washington, DC: U.S. Government Printing Office.

Newcomb, L.C., & Jerome, G.C. (1994, April). CRM

techniques applied to pilot-controller interface.

Safe application of technology in corporate aviation:

Proceedings of the 39th annual Corporate Aviation

Safety Seminar (CASS), pp. 115 - 25. Arlington,

VA: Flight Safety Foundation.

Northern NEF, Inc. (1988). Flight plan for the future:

Training controllers for the national air traffic system. (Final report delivered under OPM subcontract agreement SubOPM88-04). Colorado

Springs, CO: Author.

Norusis, M.J. (1990). SPSS Advanced statistics user’s

guide. Chicago, IL: SPSS, Inc.

Office of Deputy Associate Administrator for Appraisal.

(1992, September). An assessment of contractorprovided en route ATC training. (Report 89-2).

Washington, DC: Federal Aviation Administration Office of the Deputy Associate Administrator for Appraisal.

Office of Training and Higher Education (1989). Flight

Plan for Training. Washington, DC: Federal Aviation Administration Associate Administrator for

Human Resources Management.

Performance Verification Division. (Undated). Performance verification: TDY training guide. Oklahoma

City, OK: Federal Aviation Administration Office of Air Traffic Program Management Performance Verification Division (ATZ-400).

Rossi, P., & Freeman, H. (1985). Evaluation: A systematic approach. (3rd Ed.). Beverly Hills, CA: Sage.

Schultz, S.R., & Marshall-Mies, J. (1988). FAA ATCS

recruitment: Assessment of current procedures and

development of recommendations for an integrated

program. (HumRRO International, Inc. Technical Report 88-05 produced under OPM work

order 342-53). Alexandria, VA: Human Resources Research Organization (HumRRO) International Inc.

Seamster, T.L., Cannon, J.R., Pierce, R.M., & Redding,

R.E. (1992). Analysis of en route air traffic controller team communication and controller resource management (CRM). Proceedings of the

Human Factors Society 36th Annual Meeting, pp.

66 - 70. Santa Monica, CA: Human Factors and

Ergonomics Society.

Sherman, P.J., & Helmreich, R.L. (1993, January). The

controller resource management attitudes questionnaire (CRMAQ): Review, methodology, and results.

(NASA/UT/FAA Technical Report 93-1). Washington, DC: Federal Aviation Administration

Office of Aviation Research Air Traffic Human

Factors Program.

Smith, P.C. (1976). The problem of criteria. In M.D.

Dunnette (Ed.), Handbook of industrial/organizational psychology (pp. 745 – 75). Chicago, IL:

Rand McNally.

Stark, D.W. (1994). The validity of cognitive and noncognitive predictors over time. Unpublished doctoral dissertation, the University of Tulsa.

Thompson, M.S. (1980). Benefit-cost analysis for program evaluation. Beverly Hills, CA: Sage.

U.S. Congress. (1976, January). House committee on

government operations recommendations on air traffic

control training. Washington, DC: Author.
A-1

Appendix A 

Sample Performance Rating Packets 

A-3

Date

[OJT Instructor] 

[Facility]

[Mailing address] 

[City, ST, Zip-xxxx] 

Dear [OJT Instructor] 

The Collegiate Training Initiative for Air Traffic Control Specialists (CTI) program was implemented by the agency 

in 1990 in order to assess the feasibility of developing and implementing initial, entry-level ATCS technical training 

at selected educational institutions throughout the country. [Controller] graduated from one of the five participating 

CTI schools, and was placed directly into a field facility without going through the initial training at the FAA 

Academy. This survey is part of a series of studies being conducted by the Civil Aeromedical Institute (CAMI) to 

evaluate the feasibility of the CTI program. These survey results will be used to assess how well the CTI programs 

have trained their graduates relative to FAA-trained developmentals. 

In this evaluation, supervisors of and selected OJT instructors for the CTI graduates currently employed by the FAA 

will receive this CTI Controller Profile. In the survey, you are asked to assess the performance of [controller]--- his 

or her technical skills, technical knowledge, and teamwork --- relative to all other controllers you have known at the 

same point in their career. 

The same survey is being sent to the supervisors of and OJT instructors for a comparison group of FAA Academytrained controllers, in order to provide a basis for comparing how well the CTI and Academy programs trained new 

controllers for success in field training. Similar surveys are also being sent to the controllers to obtain their selfassessments as well. The results of the self-, supervisor-, and instructor-assessments will be combined for each 

person, and then aggregated within groups for analysis. Let us stress to you that the focus of the study is on 

evaluating the CTI and Academy programs. 

In other words, only group statistics shall be used in any reports. Data about individuals are confidential, and shall 

not be released. The data collected in this study and its results shall not be used in any way, by any member, 

employee, representative, or contractor, of the agency to effect your assignments, training, working conditions, 

or status. The surveys shall not be retained, recorded, or copied in any way at the facility for any purpose. 

Finally, the profile data from this survey are vital to evaluating the CTI program for the agency, the Department of 

Transportation, and the CTI schools. We strongly urge you to complete the survey and return it in the postage paid 

envelope as soon as possible. 

/s/

[Name] 

[Air Traffic title] 
A-4

Collegiate Training Initiative (CTI) Graduate Profile 

PURPOSE OF SURVEY

 This survey is part of a series of studies to evaluate the Collegiate Training Initiative (CTI). 

Your first-hand impressions of the performance of [controller (SSN)] are very important to 

determining if the CTI program is accomplishing the goal of putting high-aptitude persons into 

the field. 

BACKGROUND OF SURVEY

 The CTI program was initially implemented in 1990. [controller] graduated from one of the 5 

participating institutions. The purpose of this followup study is to assess the performance and 

progress of [controller] as part of the overall evaluation of the CTI. 

THANKS FOR YOUR COOPERATION

 The profile data from this survey are vital to evaluating the CTI for the agency. Your 

cooperation and thoughtful consideration of [controller]'s skills, teamwork, and knowledge is 

greatly appreciated --- and needed. 

A FINAL WORD

The researchers at CAMI know that it seems like there a lot of surveys in the FAA. This survey, 

however, has been mailed to a very specific group, as opposed to a more general sample like the 

Job Satisfaction Survey. We also know that, often, not much seems to ever be heard about the 

results of a survey. To combat this perception, we'd like to send a short summary of the results 

to you, when they become available. Just fill out a mailing label or envelope, and include it in 

the return envelope. 

THANK YOU VERY MUCH!
A-5

Please remember that this survey: 

 Is for research purposes only under 5 USC 1301, 2301, & 3304; 

 Shall be used only for the evaluation of the CTI by CAMI researchers; 

 Is confidential and only group statistics shall be used in any reports; 

 Shall not be discussed with the controller or any other person; 

 Shall not effect assignments, working conditions, or status of the controller; and 

 Shall not be copied, recorded, or retained at the facility for any purpose. 

TEAMWORK: Consider the controller's teamwork, relative to all other controllers you have observed at the 

same point in training. Use the scale below to profile [controller]'s teamwork. 

f = Exceptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at the same 

point in their career, how well does this controller: 

1. Work in a team ................................................................................. b c d e f

2. Engage other team members in solving a problem............................ b c d e f

3. Lead the team in solving problems, making improvements, etc. b c d e f

4. Earn the respect of team members..................................................... b c d e f

5. Build camadarie or spirit appropriately within the team ................... b c d e f

6. Discourage horse play or other disruptive behavior.......................... b c d e f

7. Support or aid team members in stressful situations ......................... b c d e f

8. Maintain awareness of own ability limits.......................................... b c d e f

9. Accept feedback regarding performance........................................... b c d e f

10. Seek additional information when confronted with a problem b c d e f

11. Evaluate alternative solutions to a problem....................................... b c d e f

12. Perform confidently........................................................................... b c d e f

13. Perform consistently day after day regardless of circumstances b c d e f

14. Adapt to changing conditions or circumstances on the job ............... b c d e f

15. Tolerate stressful situations ............................................................... b c d e f
A-6

f = Strongly

e = Very much

d = To some degree

c = Not very much

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at this point in 

their career, to what degree do you believe that this controller: 

16. Is part of the team.............................................................................. b c d e f

17. Might be more accepted in another work group or team................... b c d e f

18. Fits in with his or her current work group or team............................ b c d e f

19. Is treated fairly by her or his current work group or team................. b c d e f

20. Is affected negatively (with regard to work performance) by how the 

other controllers on his or her current team view her/him................. 

b c d e f

SKILLS: Now consider the controller's technical skill in performing air traffic control tasks, relative 

to all other controllers you have observed at the same point in their career. Use the scale below to 

profile [controller]'s technical skills. 

f = Execeptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at this point in 

their career, how well does this controller: 

21. Ensure separation using vectors, speed, & altitude ........................... b c d e f

22. Maintain an orderly flow of traffic.................................................... b c d e f

23. Sequence traffic................................................................................. b c d e f

24. Perform pointouts and handoffs ........................................................ b c d e f

25. Manage (surface) traffic movement areas ......................................... b c d e f

26. Recover from and correct errors, slips, and/or mistakes ................... b c d e f

27. Prioritize actions................................................................................ b c d e f

28. Maintain situational awareness.......................................................... b c d e f

29. Issue clearances using appropriate phraseology................................ b c d e f

30. Conduct relief briefing ...................................................................... b c d e f

31. Post flight data on flight progress strips............................................ b c d e f
A-7

KNOWLEDGE: Finally, consider the controller's technical knowledge about air traffic control, 

relative to all other controllers you have observed at the same point in training. Use the scale below to 

profile [controller]'s technical knowledge: 

f = Most everything

e = Much

d = Some

c = Very little

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at the same 

point in their career, how much does this controller know about: 

32. Airspace configuration in sector and/or area of specialization b c d e f

33. Traps, hot spots, and traffic patterns or flows in the sector/area b c d e f

34. Relevant sector/area LOAs and directives........................................ b c d e f

35. Relevant sector/area special procedures ........................................... b c d e f

36. ATC equipment capabilities and limitations .................................... b c d e f

37. Aircraft types, characteristics, and performance limits .................... b c d e f

38. Weather ............................................................................................ b c d e f

39. Facility general policies and procedures .......................................... b c d e f

40. Other parts of the ATC system, such as FSS and AF....................... b c d e f

41. FAA organization, general policies, and procedures........................ b c d e f

42. Pilot roles, responsibilities, constraints, and workload .................... b c d e f

Based on your observations, what is [Controller name]'s overall potential to 

succeed in the ATC occupation (on a 40-100 scale) ...............................................

About how long have you supervised or trained [controller]? .................. 

(Yrs) (Mths)

Please tell us a little about yourself: 

About how long have you been an air traffic controller? ........................................... 

About how long have you been at this facility?.......................................................... 

About how long have you been in your present position?.......................................... 

(Yrs) (Mths) 

Are you a.................. { Instructor { Supervisor { Manager 

THANK YOU VERY MUCH! 
A-8

Date

[Controller] 

[Facility]

[Mailing address] 

[City, ST, Zip-xxxx] 

Dear [controller] 

The Collegiate Training Initiative for Air Traffic Control Specialists (CTI) program was implemented by the agency 

in 1990 in order to assess the feasibility of developing and implementing initial, entry-level ATCS technical training 

at selected educational institutions throughout the country. As a graduate from one of the five participating CTI 

schools, you were placed directly into a field facility without going through the initial training at the FAA Academy. 

This survey is part of a series of studies being conducted by the Civil Aeromedical Institute (CAMI) to evaluate the 

feasibility of the CTI program. These survey results will be used to assess how well the CTI programs have trained 

their graduates relative to FAA-trained developmentals. 

In this evaluation, all of the CTI graduates currently employed by the FAA will receive this CTI Controller Profile.

In the survey, you are asked to assess your own performance --- your technical skills, your technical knowledge, 

and teamwork --- relative to all other controllers you have known at the same point in their career. We are asking 

you directly because the research literature suggests that people in fact can and do give very honest self-assessments 

when asked. People know their strengths and where they need training or practice. 

The same survey is being sent to a comparison group of FAA Academy-trained controllers, in order to provide a 

basis for comparing how well the CTI and Academy programs trained new controllers for success in field training. 

Similar surveys are also being sent to your supervisor and senior OJT instructor in order to get their assessments as 

well. The results of the self-, supervisor-, and instructor-assessments will be combined for each person, and then 

aggregated within groups for analysis. Let us stress to you that the focus of the study is on evaluating the CTI and 

Academy programs, not on individuals. 

In other words, only group statistics shall be used in any reports. Data about individuals are confidential, and shall 

not be released. The data collected in this study and its results shall not be used in any way, by any member, 

employee, representative, or contractor, of the agency to effect your assignments, training, working conditions, 

or status.

Finally, you will be given time at work to complete this survey. The profile data from this survey from each and 

every CTI graduate are vital to evaluating the CTI program for the agency, the Department of Transportation, and 

the CTI schools. While participation is voluntary, we strongly urge you to complete the survey and return it in the 

postage paid envelope as soon as possible. 

/s/ /s/ 

[Name] [Name] 

[Air Traffic title] [AHT title] 
A-9

Collegiate Training Initiative (CTI) Graduate Profile 

PURPOSE OF SURVEY

 This survey is part of a series of studies to evaluate the Collegiate Training Initiative (CTI). 

[controller], your assessment of your own performance is very important to determining if the 

CTI program is accomplishing the goal of putting high-aptitude persons into the field. 

BACKGROUND OF SURVEY

 The CTI program was initially implemented in 1990. Our records indicate that you graduated 

from one of the 5 participating institutions. The purpose of this followup study, [controller], is 

to assess the performance and progress of CTI graduates as part of the overall evaluation of the 

CTI.

THANKS FOR YOUR COOPERATION

 The profile data from this survey are vital to evaluating the CTI for the agency. Your 

cooperation and thoughtful self-assessment of your technical skills, teamwork, and knowledge is 

greatly appreciated --- and needed. 

A FINAL WORD

The researchers at CAMI know that it seems like there a lot of surveys in the FAA. This survey, 

however, has been mailed to a very specific group, as opposed to a more general sample like the 

Job Satisfaction Survey. We also know that, often, not much seems to ever be heard about the 

results of a survey. To combat this perception, we'd like to send a short summary of the results 

to you, when they become available. Just fill out a mailing label or envelope, and include it in 

the return envelope. 

THANK YOU VERY MUCH!
A-10

Please remember that this survey: 

 Is for research purposes only under 5 USC 1301, 2301, & 3304; 

 Shall be used only for the evaluation of the CTI by CAMI researchers; 

 Is confidential and only group statistics shall be used in any reports; 

 Shall not be discussed with your OJTI, supervisor, or any other person; 

 Shall not effect your assignments, working conditions, or status; and 

 Shall not be copied, recorded, or retained at the facility for any purpose. 

TEAMWORK: Consider your teamwork, relative to all other controllers you have known at the same 

point in training. Use the scale below to profile your teamwork. 

f = Exceptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have known at the same point 

in their career as you, how well do you: 

1. Work in a team ................................................................................. b c d e f

2. Engage other team members in solving a problem............................ b c d e f

3. Lead the team in solving problems, making improvements, etc. b c d e f

4. Earn the respect of team members..................................................... b c d e f

5. Build camadarie or spirit appropriately within the team ................... b c d e f

6. Discourage horse play or other disruptive behavior.......................... b c d e f

7. Support or aid team members in stressful situations ......................... b c d e f

8. Maintain awareness of own ability limits.......................................... b c d e f

9. Accept feedback regarding performance........................................... b c d e f

10. Seek additional information when confronted with a problem ......... b c d e f

11. Evaluate alternative solutions to a problem....................................... b c d e f

12. Perform confidently........................................................................... b c d e f

13. Perform consistently day after day regardless of circumstances b c d e f

14. Adapt to changing conditions or circumstances on the job ............... b c d e f

15. Tolerate stressful situations ............................................................... b c d e f
A-11

f = Strongly

e = Very much

d = To some degree

c = Not very much

b = Can't say, don't know, or not applicable

Relative to all other controllers you have known at the same point 

in their career as you, to what degree do you believe that you: 

16. Are part of the team........................................................................... b c d e f

17. Might be more accepted in another work group or team................... b c d e f

18. Fit in with your current work group or team ..................................... b c d e f

19. Are treated fairly by your current work group or team ..................... b c d e f

20. Are affected negatively (with regard to work performance) by how 

the other controllers on your current team view you......................... b c d e f

SKILLS: Now consider your technical skill in performing air traffic control tasks, relative to all other 

controllers you have known at the same point in training. Use the scale below to profile your technical 

skills.

f = Execeptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have known at the same point 

in their career as you, how well do you: 

21. Ensure separation using vectors, speed, & altitude ........................... b c d e f

22. Maintain an orderly flow of traffic.................................................... b c d e f

23. Sequence traffic................................................................................. b c d e f

24. Perform pointouts and handoffs ........................................................ b c d e f

25. Manage (surface) traffic movement areas ......................................... b c d e f

26. Recover from and correct errors, slips, and/or mistakes ................... b c d e f

27. Prioritize actions................................................................................ b c d e f

28. Maintain situational awareness.......................................................... b c d e f

29. Issue clearances using appropriate phraseology................................ b c d e f

30. Conduct relief briefing ...................................................................... b c d e f

31. Post flight data on flight progress strips............................................ b c d e f
A-12

KNOWLEDGE: Finally, consider your technical knowledge about air traffic control, relative to all 

other controllers you have known at the same point in training. Use the scale below to profile your 

technical knowledge: 

f = Most everything

e = Much

d = Some

c = Very little

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at the same 

point in their career as you, how much do you know about: 

32. Airspace configuration in sector and/or area of specialization b c d e f

33. Traps, hot spots, and traffic patterns or flows in the sector/area b c d e f

34. Relevant sector/area LOAs and directives........................................ b c d e f

35. Relevant sector/area special procedures ........................................... b c d e f

36. ATC equipment capabilities and limitations .................................... b c d e f

37. Aircraft types, characteristics, and performance limits .................... b c d e f

38. Weather ............................................................................................ b c d e f

39. Facility general policies and procedures .......................................... b c d e f

40. Other parts of the ATC system, such as FSS and AF....................... b c d e f

41. FAA organization, general policies, and procedures........................ b c d e f

42. Pilot roles, responsibilities, constraints, and workload .................... b c d e f

Based on your self-assessment, [controller], what is your overall potential to 

succeed in the ATC occupation (on a 40-100 scale) ...............................................

About how long did you have to wait between graduating and starting the 

first phase of field training at your facility?........................................................... 

(Yrs) (Mths)

THANK YOU VERY MUCH! 
A-13

Date

[Supervisor] 

[Facility]

[Mailing address] 

[City, ST, Zip-xxxx] 

Dear [supervisor] 

The Collegiate Training Initiative for Air Traffic Control Specialists (CTI) program was implemented by the agency 

in 1990 in order to assess the feasibility of developing and implementing initial, entry-level ATCS technical training 

at selected educational institutions throughout the country. [Controller] graduated from one of the five participating 

CTI schools, and was placed directly into a field facility without going through the initial training at the FAA 

Academy. This survey is part of a series of studies being conducted by the Civil Aeromedical Institute (CAMI) to 

evaluate the feasibility of the CTI program. These survey results will be used to assess how well the CTI programs 

have trained their graduates relative to FAA-trained developmentals. 

In this evaluation, supervisors of and selected OJT instructors for the CTI graduates currently employed by the FAA 

will receive this CTI Controller Profile. In the survey, you are asked to assess the performance of [controller]--- his 

or her technical skills, technical knowledge, and teamwork --- relative to all other controllers you have known at the 

same point in their career. 

The same survey is being sent to the supervisors of and OJT instructors for a comparison group of FAA Academytrained controllers, in order to provide a basis for comparing how well the CTI and Academy programs trained new 

controllers for success in field training. Similar surveys are also being sent to the controllers to obtain their selfassessments as well. The results of the self-, supervisor-, and instructor-assessments will be combined for each 

person, and then aggregated within groups for analysis. Let us stress to you that the focus of the study is on 

evaluating the CTI and Academy programs. 

In other words, only group statistics shall be used in any reports. Data about individuals are confidential, and shall 

not be released. The data collected in this study and its results shall not be used in any way, by any member, 

employee, representative, or contractor, of the agency to effect your assignments, training, working conditions, 

or status. The surveys shall not be retained, recorded, or copied in any way at the facility for any purpose. 

Finally, the profile data from this survey are vital to evaluating the CTI program for the agency, the Department of 

Transportation, and the CTI schools. We strongly urge you to complete the survey and return it in the postage paid 

envelope as soon as possible. 

/s/

[Name] 

[Air Traffic title] 
A-14

Collegiate Training Initiative (CTI) Graduate Profile 

PURPOSE OF SURVEY

 This survey is part of a series of studies to evaluate the Collegiate Training Initiative (CTI). 

Your first-hand impressions of the performance of [controller (SSN)] are very important to 

determining if the CTI program is accomplishing the goal of putting high-aptitude persons into 

the field. 

BACKGROUND OF SURVEY

 The CTI program was initially implemented in 1990. [controller] graduated from one of the 5 

participating institutions. The purpose of this followup study is to assess the performance and 

progress of [controller] as part of the overall evaluation of the CTI. 

THANKS FOR YOUR COOPERATION

 The profile data from this survey are vital to evaluating the CTI for the agency. Your 

cooperation and thoughtful consideration of [controller]'s skills, teamwork, and knowledge is 

greatly appreciated --- and needed. 

A FINAL WORD

The researchers at CAMI know that it seems like there a lot of surveys in the FAA. This survey, 

however, has been mailed to a very specific group, as opposed to a more general sample like the 

Job Satisfaction Survey. We also know that, often, not much seems to ever be heard about the 

results of a survey. To combat this perception, we'd like to send a short summary of the results 

to you, when they become available. Just fill out a mailing label or envelope, and include it in 

the return envelope. 

THANK YOU VERY MUCH! 
A-15

Please remember that this survey: 

 Is for research purposes only under 5 USC 1301, 2301, & 3304; 

 Shall be used only for the evaluation of the CTI by CAMI researchers; 

 Is confidential and only group statistics shall be used in any reports; 

 Shall not be discussed with the controller or any other person; 

 Shall not effect assignments, working conditions, or status of the controller; and 

 Shall not be copied, recorded, or retained at the facility for any purpose. 

TEAMWORK: Consider the controller's teamwork, relative to all other controllers you have observed 

at the same point in training. Use the scale below to profile [controller]'s teamwork. 

f = Exceptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at the same 

point in their career, how well does this controller: 

1. Work in a team ................................................................................. b c d e f

2. Engage other team members in solving a problem............................ b c d e f

3. Lead the team in solving problems, making improvements, etc. b c d e f

4. Earn the respect of team members..................................................... b c d e f

5. Build camadarie or spirit appropriately within the team ................... b c d e f

6. Discourage horse play or other disruptive behavior.......................... b c d e f

7. Support or aid team members in stressful situations ......................... b c d e f

8. Maintain awareness of own ability limits.......................................... b c d e f

9. Accept feedback regarding performance........................................... b c d e f

10. Seek additional information when confronted with a problem ......... b c d e f

11. Evaluate alternative solutions to a problem....................................... b c d e f

12. Perform confidently........................................................................... b c d e f

13. Perform consistently day after day regardless of circumstances b c d e f

14. Adapt to changing conditions or circumstances on the job ............... b c d e f

15. Tolerate stressful situations ............................................................... b c d e f
A-16

f = Strongly

e = Very much

d = To some degree

c = Not very much

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at this point in 

their career, to what degree do you believe that this controller: 

16. Is part of the team.............................................................................. b c d e f

17. Might be more accepted in another work group or team................... b c d e f

18. Fits in with his or her current work group or team............................ b c d e f

19. Is treated fairly by her or his current work group or team................. b c d e f

20. Is affected negatively (with regard to work performance) by how the 

other controllers on his or her current team view her/him................. 

b c d e f

SKILLS: Now consider the controller's technical skill in performing air traffic control tasks, relative 

to all other controllers you have observed at the same point in their career. Use the scale below to 

profile [controller]'s technical skills. 

f = Execeptionally

e = Very well

d = Acceptably

c = Marginally

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at this point in 

their career, how well does this controller: 

21. Ensure separation using vectors, speed, & altitude ........................... b c d e f

22. Maintain an orderly flow of traffic.................................................... b c d e f

23. Sequence traffic................................................................................. b c d e f

24. Perform pointouts and handoffs ........................................................ b c d e f

25. Manage (surface) traffic movement areas ......................................... b c d e f

26. Recover from and correct errors, slips, and/or mistakes ................... b c d e f

27. Prioritize actions................................................................................ b c d e f

28. Maintain situational awareness.......................................................... b c d e f

29. Issue clearances using appropriate phraseology................................ b c d e f

30. Conduct relief briefing ...................................................................... b c d e f

31. Post flight data on flight progress strips............................................ b c d e f
A-17

KNOWLEDGE: Finally, consider the controller's technical knowledge about air traffic control, 

relative to all other controllers you have observed at the same point in training. Use the scale below to 

profile [controller]'s technical knowledge: 

f = Most everything

e = Much

d = Some

c = Very little

b = Can't say, don't know, or not applicable

Relative to all other controllers you have observed at the same 

point in their career, how much does this controller know about: 

32. Airspace configuration in sector and/or area of specialization b c d e f

33. Traps, hot spots, and traffic patterns or flows in the sector/area b c d e f

34. Relevant sector/area LOAs and directives........................................ b c d e f

35. Relevant sector/area special procedures ........................................... b c d e f

36. ATC equipment capabilities and limitations .................................... b c d e f

37. Aircraft types, characteristics, and performance limits .................... b c d e f

38. Weather ............................................................................................ b c d e f

39. Facility general policies and procedures .......................................... b c d e f

40. Other parts of the ATC system, such as FSS and AF....................... b c d e f

41. FAA organization, general policies, and procedures........................ b c d e f

42. Pilot roles, responsibilities, constraints, and workload .................... b c d e f

Based on your observations, what is [Controller name]'s overall potential to 

succeed in the ATC occupation (on a 40-100 scale) ...............................................

About how long have you supervised or trained [controller]? ............................. 

(Yrs) (Mths)

Please tell us a little about yourself: 

About how long have you been an air traffic controller? ........................................... 

About how long have you been at this facility?.......................................................... 

About how long have you been in your present position?.......................................... 

(Yrs) (Mths) 

Are you a.................. { Instructor { Supervisor { Manager 

THANK YOU VERY MUCH! 

B-1

Appendix B 

Cost-Benefit Analysis Model Worksheets 

B-3

Table B-1 

FY1991 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $6,650,000 $6,650,000 

 Direct Site Visits 6.00 $500 $3,000 

 Direct Regional Liaison (GS-14/5) 1.00 $59,394 $59,394 

 Program AHT Staff (GS-14/5) 0.25 $59,394 $14,849 

 Program AAM Psychologist (GS-12/1) 0.10 $37,294 $3,729 

 Program Evaluation Contract(s) 0.00 $0 $0 

 Program APN Staff (GS-13/5) 0.00 $50,260 $0 

 Program AMH Staff (GS-7/2) 0.00 $21,724 $0 

 Program AMA Staff (GS-14/5) 0.01 $59,394 $594 

 Program ATZ Staff (GS-14/5) 0.04 $59,394 $2,376 

 Annual Costs $6,733,942 

 Cumulative Costs $6,733,942 

 Annual Hires 26 

 Cumulative Hires 26 

 Cost per Hire $258,998 

Table B-2 

FY1992 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $2,000,000 $2,000,000 

 Direct Site Visits 1.00 $500 $500 

 Direct Regional Liaison (GS-14/5) 0.00 $61,887 $0 

 Program AHT Staff (GS-14/5) 0.05 $61,887 $3,094 

 Program AAM Psychologist (GS-12/2) 0.10 $40,156 $4,016 

 Program Evaluation Contract(s) 0.00 $0 $0 

 Program APN Staff (GS-13/9) 0.10 $58,530 $5,853 

 Program AMH Staff (GS-7/3) 0.00 $23,366 $0 

 Program AMA Staff (GS-14/5) 0.01 $61,887 $619 

 Program ATZ Staff (GS-14/5) 0.04 $61,887 $2,475 

 Annual Costs $2,016,557 

 Cumulative Costs $8,750,499 

 Annual Hires 19 

 Cumulative Hires 45 

 Cost per Hire $194,456 
B-4

Table B-3 

FY1993 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $2,000,000 $2,000,000 

 Direct Site Visits 1.00 $500 $500 

 Direct Regional Liaison (GS-14/5) 0.50 $64,179 $32,090 

 Program AHT Staff (GS-14/5) 0.04 $64,179 $2,567 

 Program AAM Psychologist (GS-13/1) 0.02 $47,920 $958 

 Program Evaluation Contract(s) 0.20 $97,000 $19,400 

 Program APN Staff (GS-13/9) 0.07 $60,696 $4,249 

 Program AMH Staff (GS-7/4) 0.12 $24,988 $2,999 

 Program AMA Staff (GS-14/5) 0.01 $64,179 $642 

 Program ATZ Staff (GS-14/5) 0.01 $64,179 $642 

Annual Costs $2,064,046 

 Cumulative Costs $10,814,545 

 Annual Hires 78 

 Cumulative Hires 123 

 Cost per Hire $87,923 

Table B-4 

FY1994 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $0 $0 

 Direct Site Visits 1.00 $500 $500 

 Direct Regional Liaison (GS-14/5) 0.01 $64,179 $642 

 Program AHT Staff (GS-14/6)* 0.10 $68,862 $6,886 

 Program AAM Psychologist (GS-13/2) 0.10 $51,171 $5,117 

 Program Evaluation Contract(s) 0.00 $0 $0 

 Program APN Staff (GS-13/10)* 0.10 $64,928 $6,493 

 Program AMH Staff (GS-7/4) 0.12 $25,823 $3,099 

 Program AMA Staff (GS-14/5) 0.01 $66,323 $663 

 Program ATZ Staff (GS-14/5)* 0.02 $66,323 $1,326 

*Locality pay Annual Costs $24,726 

 Cumulative Costs $10,839,271 

 Annual Hires 40 

 Cumulative Hires 163 

 Cost per Hire $66,499 
B-5

Table B-5 

FY1995 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $0 $0 

 Direct Site Visits 0.00 $2,000 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/6)* 0.02 $71,078 $1,422 

 Program AAM Psychologist (GS-13/3)* 0.10 $52,136 $5,214 

 Program Evaluation Contract(s) 0.00 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/5)* 0.12 $26,529 $3,183 

 Program AMA Staff (GS-14/5)* 0.01 $65,460 $655 

 Program ATZ STaff (GS-14/5)* 0.02 $69,047 $1,381 

*Locality pay Annual Costs $15,860 

 Cumulative Costs $10,855,131 

 Annual Hires 30 

 Cumulative Hires 193 

 Cost per Hire $56,244 

Table B-6 

FY1996 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $250,000 $250,000 

 Direct Site Visits 0.00 $0 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/6)* 0.05 $71,078 $3,554 

 Program AAM Psychologist (GS-14/1)* 0.10 $57,760 $5,776 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/5)* 0.12 $26,529 $3,183 

 Program AMA Staff (GS-14/5)* 0.01 $65,460 $655 

 Program ATZ Staff (GS-14/5)* 0.02 $69,047 $1,381 

*Locality pay Annual Costs $268,555 

 Cumulative Costs $11,123,686 

 Annual Hires 0 

 Cumulative Hires 193 

 Cost per Hire $57,636 
B-6

Table B-7 

FY1997 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,700,000 $1,700,000 

 Direct Site Visits 0.00 $0 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/7)* 0.05 $73,108 $3,655 

 Program AAM Psychologist (GS-14/2)* 0.10 $59,685 $5,969 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/6)* 0.12 $27,031 $3,244 

 Program AMA Staff (GS-14/6)* 0.01 $67,385 $674 

 Program ATZ Staff (GS-14/6)* 0.02 $71,078 $1,422 

*Locality pay Annual Costs $1,718,969 

 Cumulative Costs $12,842,654 

 Annual Hires 64 

 Cumulative Hires 257 

 Cost per Hire $49,971 

Table B-8 

FY1998 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $0 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/7)* 0.05 $73,108 $3,655 

 Program AAM Psychologist (GS-14/2)* 0.10 $59,685 $5,969 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/6)* 0.12 $27,031 $3,244 

 Program AMA Staff (GS-14/6)* 0.01 $67,385 $674 

 Program ATZ Staff (GS-14/6)* 0.02 $71,078 $1,422 

*Locality pay Annual Costs $1,518,969 

 Cumulative Costs $14,361,623 

 Annual Hires 100 

 Cumulative Hires 357 

 Cost per Hire $40,229 
B-7

Table B-9 

FY1999 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $500 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/8)* 0.05 $75,139 $3,757 

 Program AAM Psychologist (GS-14/3)* 0.10 $61,610 $6,161 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/7)* 0.12 $27,803 $3,336 

 Program AMA Staff (GS-14/7)* 0.01 $69,310 $693 

 Program ATZ Staff (GS-14/7)* 0.02 $73,108 $1,462 

*Locality pay Annual Costs $1,519,415 

 Cumulative Costs $15,881,038 

 Annual Hires 100 

 Cumulative Hires 457 

 Cost per Hire $34,751 

Table B-10 

FY2000 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $500 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/8)* 0.05 $75,139 $3,757 

 Program AAM Psychologist (GS-14/3)* 0.10 $61,610 $6,161 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/7)* 0.12 $27,803 $3,336 

 Program AMA Staff (GS-14/7)* 0.01 $69,310 $693 

 Program ATZ Staff (GS-14/7)* 0.02 $73,108 $1,462 

*Locality pay Annual Costs $1,519,415 

 Cumulative Costs $17,400,453 

 Annual Hires 100 

 Cumulative Hires 557 

 Cost per Hire $31,240 
B-8

Table B-11 

FY2001 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $500 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/8)* 0.05 $75,139 $3,757 

 Program AAM Psychologist (GS-14/3)* 0.10 $61,610 $6,161 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/7)* 0.12 $27,803 $3,336 

 Program AMA Staff (GS-14/7)* 0.01 $69,310 $693 

 Program ATZ Staff (GS-14/7)* 0.02 $73,108 $1,462 

*Locality pay Annual Costs $1,519,415 

 Cumulative Costs $18,919,869 

 Annual Hires 100 

 Cumulative Hires 657 

 Cost per Hire $28,797 

Table B-12 

FY2002 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $500 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/8)* 0.05 $75,139 $3,757 

 Program AAM Psychologist (GS-14/3)* 0.10 $61,610 $6,161 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/7)* 0.12 $27,803 $3,336 

 Program AMA Staff (GS-14/7)* 0.01 $69,310 $693 

 Program ATZ Staff (GS-14/7)* 0.02 $73,108 $1,462 

*Locality pay Annual Costs $1,519,415 

 Cumulative Costs $20,439,284 

 Annual Hires 100 

 Cumulative Hires 757 

 Cost per Hire $27,000 
B-9

Table B-13 

FY2003 MnATCTC Cost estimate worksheet

 Cost Category Description N Rate $ 

 Direct FAA Financial Support 1.00 $1,500,000 $1,500,000 

 Direct Site Visits 0.00 $500 $0 

 Direct Regional Liaison (GS-14/5) 0.01 $65,460 $655 

 Program AHR-15 Staff (GS-14/8)* 0.05 $75,139 $3,757 

 Program AAM Psychologist (GS-14/3)* 0.10 $61,610 $6,161 

 Program Evaluation Contract(s) 0.20 $0 $0 

 Program AHR-22 Staff (GS-13/10)* 0.05 $67,021 $3,351 

 Program AMH Staff (GS-7/7)* 0.12 $27,803 $3,336 

 Program AMA Staff (GS-14/7)* 0.01 $69,310 $693 

 Program ATZ Staff (GS-14/7)* 0.02 $73,108 $1,462 

*Locality pay Annual Costs $1,519,415 

 Cumulative Costs $21,958,699 

 Annual Hires 100 

 Cumulative Hires 857 

 Cost per Hire $25,623 
B-10

Table B-14 Overall cost-benefit estimate worksheet MnATCTC hires by FAA FAA Screening FAA Academy 

FY Annual Cumulative Screening cost 

Cumulative 

avoided costs 

Academics 

(Phase I) cost 

Techniques 

(Phase II) cost 

Skills-building 

(Phase III) cost 

Cumulative 

avoided costs 

1991 26 26 $10,000 $260,000 1992 19 45 $10,000 $450,000 1993 78 123 $1,500 $567,000 $15,000 $15,000 $15,500 $1,170,000 1994 40 163 $1,500 $627,000 $15,000 $15,000 $15,500 $2,990,000 1995 30 193 $1,500 $672,000 $15,000 $15,000 $15,500 $4,355,000 1996 0 193 $1,500 $672,000 $15,000 $15,000 $15,500 $4,355,000 1997 64 257 $0 $672,000 $15,000 $15,000 $15,500 $7,267,000 1998 100 357 $0 $672,000 $15,000 $15,000 $15,500 $11,817,000 1999 100 457 $0 $672,000 $15,000 $15,000 $15,500 $16,367,000 2000 100 557 $0 $672,000 $15,000 $15,000 $15,500 $20,917,000 2001 100 657 $0 $672,000 $15,000 $15,000 $15,500 $25,467,000 2002 100 757 $0 $672,000 $15,000 $15,000 $15,500 $30,017,000 2003 100 857 $0 $672,000 $15,000 $15,000 $15,500 $34,567,000 

(Table B-14 continues across next page) 
B-11

(Table B-14 continues across page) 

 Performance verification On-the-job training 

FY PV Cost 

Cumulative 

avoided cost 

 Reduction in 

years to FPL 

Average OJT 

cost per year

Annual avoided 

cost

Cumulative avoided 

costs

Total cumulative 

avoided FAA costs

Cumulative 

MnATCTC Costs

Benefit to 

cost ratio 

1991 0.07 $43,913 $3,074 $79,922 $339,922 $6,733,942 $0.05 1992 0.07 $43,913 $3,074 $138,326 $588,326 $8,750,499 $0.07 1993 $500 $39,000 0.07 $43,913 $3,074 $378,091 $2,115,091 $10,814,545 $0.20 1994 $500 $59,000 0.07 $43,913 $3,074 $501,047 $4,118,047 $10,839,271 $0.38 1995 $500 $74,000 0.07 $43,913 $3,074 $593,265 $5,620,265 $10,855,131 $0.52 1996 $500 $74,000 0.07 $43,913 $3,074 $593,265 $5,620,265 $11,123,686 $0.51 1997 $500 $106,000 0.07 $43,913 $3,074 $789,995 $8,728,995 $12,842,654 $0.68 1998 $500 $156,000 0.07 $43,913 $3,074 $1,097,386 $13,586,386 $14,361,623 $0.95 1999 $500 $206,000 0.07 $43,913 $3,074 $1,404,777 $18,443,777 $15,881,038 $1.16 2000 $500 $256,000 0.07 $43,913 $3,074 $1,712,168 $23,301,168 $17,400,453 $1.34 2001 $500 $306,000 0.07 $43,913 $3,074 $2,019,559 $28,158,559 $18,919,869 $1.49 2002 $500 $356,000 0.07 $43,913 $3,074 $2,326,950 $33,015,950 $20,439,284 $1.62 2003 $500 $406,000 0.07 $43,913 $3,074 $2,634,341 $37,873,341 $21,958,699 $1.72