i i iii

DOT/ F~AAAM-90/13 Selection of Air Traffic

Controllers for

Automated Systems:

Applications from

Office of Aviation in~o, Medicine 0 1Current Research

Washington, D.C. 20591 C r e tR s a c

Pamela S. Della Rocco, M.S.

' -" Carol A. Manning, Ph.D.

Civil Aeromedical Institute o -t 199 Federal Aviation Administration

Oklahoma City, OK 73125

Hilda Wing, Ph.D.

Staffing Policy Division

Federal Aviation Administration

Washington, D.C. 20591

November 1990

: - Final Report

This document is available to the public through

the National Technical Information Service,

Springfield, Virginia 22161.

U.S. Department

of Transportation

Federal Aviation

Admlnlbtratlon
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents or use thereof.
Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-90/13I

4. Title and Subttite 5. Repo.- Date

Selection of Air Traffic Controllers for Automated November 1990

Systems: Applications From Current Research 6 Performing Organi za on Code

7 8. Performing Organ, zaton Report No. 7. Autkor'

Pamela S. Della Rocco, Carol A. Manning, Hilda Wing

9. Perforrmng Organzat,on Name and Address 10. Work Unit No. ,TRAIS)

FAA Civil Aeromedical Institute

P. 0. Box 25082 11. Contract or Grant No.

Oklahoma Oklahoma------- City, OK 73125[3 Te 513. Type of Report and Period Covered

12. Sponsoring Agency Name and Address

Office of Aviation Medicine

Federal Aviation Administration

800 Independence Ave., S.W. 14. Sponsorring Agency Code

Washington, D. C. 20591

1 15 Supplementary Notes

This task was performed under task AM-C-90-HRR-123.

16. Abstract

Over the next two decades, the Federal Aviation Administration's (FAA) plan for new

automated systems will change the air traffic control specialist's (ATCS) job as many

of the current controller's tasks become automated. The purpose of this paper was

to review the findings from current research on selection of ATCS's that may guide

the design of selection systems for future controllers. To accomplish this two lines

of research were presented: 1) projected changes in job tasks resulting from

planned automation, and 2) the current ATCS selection system. A study completed

in 1987 estimated that 48 of 337 job tasks of the enroute controller would be

substantially changed with implementation of the Initial Sector Suites (ISSS). In light

of the projected changes, the current selection system was evaluated in terms of the

methodologies used for selection and the utility and validity of those methodologies.

The current job is a highly complex set of tasks and demands high levels and active

application of certain cognitive abilities, such as spatial perception, information

processing, reasoning and decision making. Evaluation of the changes projected in

the job over the next two decades suggested that a similar performance-based

selection system could maintain utility through implementation of the ISSS--

However, implementation of the more advanced automation may significantly change

the cognitive skills and abilities required for successful performance. Thus, work

toward selection for the advanced automated environment should begin immediately.

17. Key W ards 18. Dristrbution Statement.

Air Traffic Controller Specialists ." V"

Personnel Selection

Advanced Automation

AERA c

19. Security Class,f. tof this report) 20. Securty Class hS p ge 21. No, 2

Unclassified Unclassified 38 J

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

... mmm n.,.., Jmmmlnmim • II i n mI
CONTENTS

Introduction ............................................................................................................................... 1

Autom ation and Air Traffic Control Services ............................................................. 2

NAS Plan and the Air Traffic Control Specialist ............................................................ 3

ATC Job Task Analyses ..................................................................................................... 4

Affect of ISSS and AERA on Job Tasks ......................................................................... 6

- The Initial Sector Suite System ........................................................................................ 6

- Automated En Route Air Traffic Control (AERA) ........................................................ 7

Current Selection of Air Traffic Control Specialists ............................................... 8

Personnel Selection in the United States ......................................................................... 8

Selection of Air Traffic Controllers .................................................................................. 11

- History of the Development of ATCS Selection

Procedures ........................................................................................................................... 11

- The Current ATCS Selection System .......................................................................... 13

- The ATCS Selection Procedures and Field Training

Status and Attrition Rates ............................................................................................... 15

- Current Correlations Between Performance on the

OPM Battery, Academy Screen, and Field Training

S ta tu s ................................................................................................................................... 1 5

- Use of Nonradar ATC Tasks to Predict a RadarBased Job .............................................................................................................................. 19

Computerized Selection Systems .................................................................................. 20

Future Selection System s ............................................................................................ 20

Discussion ............................................................................................................................... 22

References ................................................................................................................................ 24

Accesion For

NTIS CRA&I 4 1

DI IC TAB

Di t ib :o',l

ib i " ll;h0,y C'o
LIST OF TABLES/APPENDIX

Table 1: Correlations between OPM and Academy Screen

Performance Measures and Measures of Field Training

S tatus .............................................................................................. . . 17

Table 2: Correlations between OPM and Screen Performance

Measures and Measures of Field Training Status ..................... 18

Appendix A: Cognitive/Sensory Attribute Definitions from Computer

Technology Associates' Job Task Analysis ................................... A 1

Appendix B: Applicant OPM ATCS Battery Performance .............................. B1

Appendix C: Academy Entrant OPM ATCS Battery Performance ................ C1

Appendix D: Academy Performance Measures ................................................ D1

Appendix E: Correlations between Academy and OPM Battery

Com ponent Scores ......................................................................... El

V.
GLOSSARY OF ABBREVIATIONS

AAS - Advanced Automation System

ACF - Area Control Facility AERA - Automated En Route Air Traffic Control

ARTCC - Air Route Traffic Control Center

ATACT - Air Traffic AERA Concepts Team

ATCS - Air Traffic Control Specialist

CAA - Civil Aeronautics Administration

CAMI - Civil Aeromedical Institute

CSC - Civil Service Commission

CST - Controller Skills Test

CTA - Computer Technology Associates, Inc.

FAA - Federal Aviation Administration

FPL - Full Performance Level

GPS - Globai Positioning System ISSS - Initial Sector Suite System

JTA - Job Task Analysis

KSA - Knowledge, Skills, and Abilities

MCAT - Multiplex Controller Aptitude Test

NAS - National Airspace System NOTAM - Notices to Airmen

OKT - Occupational Knowledge Test

OJT - On The Job Training

OPM - Office of Personnel Management PVD - Plan View Display

SME - Subject Matter Experts

VFR - Visual Flight Rules

vii.
SELECTION OF AIR TRAFFIC CONTROLLERS FOR AUTOMATED SYSTEMS:

APPLICATIONS FROM TODAY'S RESEARCH

INTRODUCTION

Over the next two decades, the Federal Aviation Administration's (FAA) National

Airspace System (NAS) Plan for new automated air traffic control systems will

radically change the job of Air Traffic Control Specialist (ATCS). While the

introduction of automation is not new to the occupation, the NAS Flan calls for

automating more critical job tasks than could have been previously supported by

technology. The challenge of selecting the controller* to operate these future

automated systems is currently facing the FAA.

The first step in designing a selection system for current or future jobs is an

analysis of the tasks required to perform the job. In approaching such analysis, a

would-be designer of a selection system unfamiliar with air traffic control faces a

challenge. Hopkin (1) has captured first impressions of the job in the following

comments:

"To a naive observer, the workspace of controllers seems complex and well

nigh meaningless. It is not intuitively and immediately obvious what any of

the data mean, what the tasks are, how they are done, or what would

constitute successful performance."

Thus, the first step toward a successful selection system involves analysis of tasks

not immediately obvious and perhaps not easily measured.

By definition, a controller is tasked with promoting the safe, orderly and expeditious

flow of air traffic. This is accomplished through accurate, effective application of

rules and procedures in a real-time, dynamic environment. The current ATCS's job

consists of a complex set of tasks that demand a high degree of skill and active

application of certain cognitive abilities, such as spatial perception, information

processing, reasoning, and decision making. As an increasing number of ATC tasks

become automated, the controller's participation may gradually evolve from an active

manager to a more passive system monitor.

*The terms ATCS and controller are used interchangeably throughout this paper.

ATCS is the formal job title, whereas "controller" is the more familiar, informal title.

, . | , l I I II I i

I 1
The current work force has been rigorously selected through a two stage selection

system. The first stage consists of a paper-and-pencil aptitude test battery. The

second stage is a performance-based measure consisting of a condensed trainingtesting ATC task sample in a nonradar environment. As the job itse!f changes,

questions arise as to how long a selection system based upon the current selection

criteria will be effective, and what factors will be important in designing the

replacement.

To address these questions, the job of the ATCS in its current form will be reviewed

and contrasted to projected changes in job tasks which will result from

implementation of certain components of the NAS Plan; specifically the Initial Sector

Suite System (ISSS) and the Automated En Route Air Traffic Control (AERA)

software enhancements of the Advanced Automation System (AAS). The current

selection system will then be examined to determine the extent to which the current

selection criteria might predict success in future ATC jobs as automation causes the

tasks to evolve.

AUTOMATION AND AIR TRAFFIC CONTROL SERVICES

Over the 50-year history of the ATC system's evolution, a mixture of equipment of

varying technological generations and types was used. Computer automation

techniques were first applied to flight data processing systems utilizing UNIVAC and

IBM computers during the early 1960s. The systems were installed in six

northeastern Air Route Traffic Control Centers (ARTCCs). Through this effort the

FAA demonstrated that computers could be used to improve aviation safety and

increase the controller's productivity. In 1968 an integrated plan to automate ATC

functions was approved. This plan was NAS Stage A. By 1981 the NAS Plan for

Facilities, Equipment and Associated Development was issued to define an orderly

rational evolution of the system as a whole. The plan has been updated annually.

The most recent version is the NAS Plan: Facilities, Equipment and Associated

Development, and other Capital Needs, which was issued in September 1989 (2).

The NAS Plan established three principal objectives for the en route and terminal

ATC systems. The first was consolidation of more than 200 facilities existing today

into less than 30 by the year 2000. The second objective was to install common

modular computers, software, and controller work stations (i.e., "Sector Suites"). To

meet these two objectives, the en route and terminal radar approach control facilities

will be consolidated into area control facilities (ACF's) with essentially identical

equipment, including the HOST computer and the Initial Sector Suite System (ISSS).

The third objective was the improvement of safety, fuel efficiency and productivity

through higher levels of automation. The NAS Plan includes improvements such as

Mode S data link, ASR-9 and advanced Doppler weather radar, navigation by Global

Positioning System (GPS) satellites, and advanced stages of automation software,

2
such as Automated En Route Air Traffic Control (AERA). Other advances in the air

traffic system that will affect the controllers' functions include point-to-point navigation

and direct routing from point of departure to destination.

The NAS Plan provides for a staged delivery of the various equipment and software

developments. The HOST computers have been installed and facilities await

delivery of the first Sector Suites beginning in the mid-1990s.

NAS PLAN AND THE AIR TRAFFIC CONTROL SPECIALIST

Two automation advancements in the NAS Plan will particularly affect the

environment and job tasks of the en route controller: Sector Suites and the

implementation of AERA.

The Sector Suites provide an entirely new controller workstation. Briefly described,

each Sector Suite is a collection of one to four common consoles, each of which

contain data entry and display equipment. The Sector Suite consoles will provide

information to the controller on multiple display screens. A common console will

consist of a main and an auxiliary display, voice switching, and control panels.

Display screens will have multi-color capability. Data entry devices will include a

keyboard and trackball. Information presented will include a situation display of the

air traffic and weather, alphanumeric flight and weather data, as well as other

pertinent information, such as Notices to Airmen (NOTAM's). The most substantial

change to the controller's environment presented by the Sector Suites is the

replacement of paper flight progress strips with an electronic display of flight data

(3).

AERA will also substantially affect the en route controller's job tasks by developing a

software system, which will enhance the earlier ATC hardware and software system.

AERA will support the ATCS by predicting and resolving prcblems along an

aircraft's route of flight, and by planning and maintaining traffic flow and aircraft

separation.

The NAS Plan proposes three stages of AERA software enhancements: AERA 1,

AERA 2, and AERA 3. Under AERA 1, the computer software will detect certain

common problems, such as conflicts between aircraft, violations of protected

airspace, and non-adherence to ATC-imposed traffic flow restrictions. AERA 2 will

then provide enhanced assistance to controllers through proposed resolution of

detected problems. Several resolutions for a problem will be generated. The

highest ranked resolution will be presented to the controller currently controlling the

aircraft affected by the resolution. The controller will assess that resolution, examine

others if necessary, and determine which resolution to choose, depending on his or

her requirements for the airspace. This activity will be complicated by the increased

use of direct routings instead of currently established jetways. For aircraft equipped

with data link, clearances will be transmitted by the computer, once approved by the

3
controller. Clearances for aircraft without data link will be delivered by the controller

through a voice channel. Certain coordination functions between controllers will also

be automated. The controller remains responsible for all control actions with AERA

2. However, with AERA 3, automated decision-making and control will be

introduced with a certain degree of autonomy for the computer to detect and resolve

ATC problems without human intervention.

The NAS Plan establishes a clear path toward increased automation of air traffic

control. If the current responsibility of the controller is to ensure the separation cf

aircraft a3 well as the safe, expeditious flow of traffic, then by the time AERA 3 is

imolemented, much of that function could shift to the computer. To design a

selection system for the evolving job of the controller or to assess aspects of the

current selection system that might have utility for the future, it is important to

understand the job tasks of the controller and the changes that will be made in

those tasks with increased automation. The following section examines the current

functions of an ATCS and reviews studies analyzing the impact of Initial Sector

Suite System and Automated En Route Air Traffic Control on those functions.

ATC JOB TASK ANALYSES

A job task analysis (JTA) provides a detailed itemization of the tasks required by a

specific job. Such a detailed analysis can help us systematically map changes in

job functions with each phase of the NAS Plan. From there we can begin to assess

the knowledge, skills, and abilities required of the human operator. In addition, the

JTA can help us understand what we are accomplishing with current selection

systems.

In preparation for the transition to the Advanced Automation System (AAS) under

the NAS Plan, Computer Technology Associates, Inc. (CTA) was commissioned to

conduct a job task analysis in today's ATC environment, as well as for the planned

automat!on capabilities. The result of this effort was a multi-volume documentation

of Operations Concepts for ATC personnel in operational environments at different

stages of AAS evc!ution. The focus was placed on the interaction between the

controller and the automated system during operational tasks as described in

Volume 1 (4).

Data were collected through field observations, interviews conducted in a large cross-section of ATC facilities and through regular involvement of a controller

validation ream. Thus, the JTA was completed and validated with substantial input

from subject matter experts (i.e., ATCSs).

The CTA methodology derived a task list utilizing a stimulus-process-response model

in which a controller performs actions in response to air traffic events. Therefore,

the job analysis began with identifying and defining the air traffic events to which a

controller responds. After identifying the response, the controller's action was then

4
decomposed to the task level--the central analysis result. Within this framework, the

work performance actions were classified into three increasingly detailed levels:

activities, sub-activities, and tasks. Activities were the highest level of description of

operational job functions. The operational duties of a position were typically

described with five to nine activities. Sub-activities were a more detailed statement

of the work performed, and a task was the most detailed unit of description.

Within the task level specification, four task types were identified as follows: 1)

ENTRY tasks (i.e, entry of data into the system), 2) RECEIPT tasks (e.g. receipt of

information from the displays), 3) ANALYTICAL tasks, and 4) VERBAL

COORDINATION tasks (coordination with other individuals or the system).

Task characterization analyses were also performed that described tasks on four

separate dimensions. Two of these dimensions are of particular interest in

establishing a controller selection system. These two task characterizations are 1)

the cognitive/sensory attributes required for a task and 2) the identification of task

perforrance requirements.

For purposes of this paper, only the tasks required of the controllers working in Air

Route Traffic Control Centers (ARTCCs)--the en route controllers--will be reviewed.

These are described in Volume VI of the Operations Concepts documentation (5).

Six activities were identified for the en route controller in the current ATC

environment, as follows:

1) perform situation monitoring;

2) resolve aircraft conflicts;

3) manage air traffic sequences;

4) route or plan flights;

5) assess weather impact;

6) manage sector/position resources.

Within the 6 activities, 46 sub-activities and a total of 348 distinct tasks were

identified. Clearance delivery was identified as a set of tasks that was relevant to

many sub-activities and was thus defined as a "macro" performed as a function of a

number of tasks.

The cognitive/sensory attributes re"uired for performing tasks of extreme or high

criticality were identified. Fo:.rteen attributes were found to be relevant to

performance of computer and radar workstation jobs associated with current ATC:

coding, movement detection, spatial scanning, filtering, image/pattern recognition,

decoding, visualization, sK-rt-term memory, long-term memory, deductive reasoning,

inductive reasoning, mathematical/probabilistic reasoning, prioritizing, and verbal

filtering. (Definitions for these attributes are included in Appendix A).

Further Analysis of each of 161 critical tasks revealed that analytical attributes such

as visualization (42 tasks), short-term memory (30 tasks), deductive reasoning (41

5
tasks), and mathematical/probabilistic reasoning (32 tasks), were k6y cognitive attributes. Additionally, message filtering (40 tasks) and decoding (57 tasks) were

important.

AFFECT OF ISSS AND AERA ON JOB TASKS

The Initial Sector Suite System

In 1988, Computer Technology Associates completed a report of the impact of the

Initial Sector Suite System on ATC procedures and training (6). Expanding on a

prior report, the analysis provided an early identification of current ATC procedures

projected to be affected by ISSS. The analysis for the report compared the

Operations Concepts for the current job, described above, and the Operations Concepts preoared for ISSS utilizing the same job analysis methodologies (7). The

study estimated that 48 of 373 (13%) job tasks of the en route controller would be

substantially changed from the current ATC system; 237 (64%) would be essentially

unchanged. The remaining 88 tasks would be affected to some degree, but not

substantially changed.

CTA observes that the ISSS introduces substantial changes to the en route

controller's operations. In general, the ISSS will provide the controller with greater flexibility and capability in data handling. There will be increased data processing

and flight data will be available more rapidly. Entry of data into the system will be

increased over today's requirements.

While a comparison of information displayed reveals a distinct parallelism between

current sector workstation physical displays and written records and the Sector Suite

workstation, the manner in which controllers manipulate the data will be significantly

changed. The most substantial change with ISSS is the conversion of paper flight

progress strips to electronic flight data entry and electronic display and manipulation

of the data. Today's flight progress strips are manually inserted into plastic holders, placed in sequence in a strip bay, marked with a pen, sorted, flagged, and removed.

Under ISSS all of these actions will be accomplished electronically. Computer Technology Associates notes that display management will be much more critical

with ISSS than with today's system, because more information will be available

under ISSS and will need to be displayed for most efficient use. CTA also

expresses a concern that electronic presentation of data may impact the silent

coordination between the radar and radar associate ("D-side") controllers.

In summary, the 48 radically and significantly different tasks under ISSS pertain to

the controllers' interaction with information displays and the input of control data and

messages. CTA's analysis of changes in the cognitive/sensory attributes of these

tasks determined that "coding" will be substantially increased because of the

expanded availability of messages. CTA concluded that cognitive and sensory

6
attributes required for the more critical controller tasks in both the current ATC

system and under ISSS would be similar. In terms of performance requirements,

the accuracy of data entry would become critical and the frequency of receipt of

information from the system would increase.

Automated En Route Air Traffic Control (AERA)

Researcriers at MITRE, in conjunction with teams of controllers as subject matter

experts, prepared several papers assessing the changes in the en route controllers'

operational functions after implementation of AERA. AERA 1 introduces primarily

problem detection capability and some decision aids for the en route controller.

However, AERA 2 expands these capabilities in several important areas. These

areas include resolutions to problems detected by the automated problem detection

software, display of proposed resolutions to the appropriate controller, and finally

provision of additional aids to assist a controller in decision-making for separation of

air traffic and coordination.

Carlson and Rhodes (8) described the changes in the controllers' work activities in

AERA 2. In today's environment, controllers rely on their own mental abilities to

detect and resolve problems occurring in their airspace. ATCSs must recognize

potential loss of separation between aircraft based upon available data in flight

progress strips and plan view displays (situation displays) showing computer

processed radar returns of current and recent past aircraft positions, and their

knowledge about the operational ATC environment. Upon recognizing a potential

problem, the ATCS must formulate and evaluate resolutions for the situation. The

human being has a finite capacity to process information. Limits include the

maximum number of aircraft an invididL' '

i can safely and efficiently control at any

one time, as well as the look-ahead tir 4 to project potential conflicts. Introduction

of direct routing of aircraft and reduced use of established airways may, in fact,

overload a controller's information processing capabilities.

AERA 2 should expand the capacity of the ,TC system by automating the detection

of problems up to 20 minutes prior to predicted loss of separation and for an

aircraft's entire route of flight. The Automated Problem Resolution (APR) function

will produce a number of resolutions based upon the aircraft's projected trajectory,

geometry of the conflict, airspace characteristics, and other aircraft or system

requirements. This capability is projected to improve detection and resolution of

problems because they will be detected earlier, allowing more optimal resolutions to

be considered.

The affect of AERA 2 on the controller's job has been described by Carlson and

Rhodes (8) as shifting the emphasis away from tactical decision-making to more

strategic management of the ATC environment. They suggest that many routine

tasks will be automated, and more complex tasks will become easier because of the

availability of computer proposed resolutions, suggesting that controllers will rely less

on their own mental abilities to monitor flight data and identify situations requiring

7
control actions. The focus of the controller's attention will shift from the flight data

and situation display to the problem alert and resolution displays. Finally, they

suggest that because problems will be resolved earlier than they are in today's

system, the controller's time orientation for detecting and resolving ATC problems

will change.

Another perspective on the time orientation of the controller under AERA was

provided by McKinley and Jago (9). In an early evaluation of the en route

controller's skills requirements, they suggested that tne automated control capability

of AERA would only be possible for long-term (greater than 20 minutes in the

future) and some medium-term (6-20 minutes) problem prediction and resolution.

For the short-term (0-5 minutes) and some medium-term problem detection, reliance

on the information from the situation display and electronic flight progress data

would continue to be required.

Because AERA 2 and AERA 3 are still under development, it is difficult to assess

the cognitive skills and abilities required for the job tasks. However, it is clear that

the implementation of AERA 2, and particularly AERA 3, will significantly affect the

role of the ATCS in controlling traffic and that research on the impact must continue.

CURRENT SELECTION OF AIR TRAFFIC CONTROL SPECIALISTS

PERSONNEL SELECTION IN THE UNITED STATES

The discussion of selection of controllers should be considered within the context of

selection of personnel, in general, in the United States. That context is highly

regulated, considerably politicized, and normally very public. Prior to the Civil Rights

Act of 1964, selection was a backwater of personnel psychology, i-oluding only a

few individuals of talent and senstivity among its ranks. Civil rights activists

perceived, mostly accurately perhaps, that the selection procedures fo!!owed by

many companies bore little relationship to the jobs at hand while they excluded high

percentages of minority applicants. Much of the early case law on personnel

selection was built from such cases. It could be argued that many of the resulting

judicial opinions overgeneralized and, by inference, developed procedural

requirements that few, if any, of the best selection systems could meet. Since the

1960s, selection procedures have become the subject of scrutiny by many different

groups. This has meant that the procedures must be both extensively documented

and technically acceptable. Because of this, personnel psychology left its position in

the backwaters and has become one of the more exciting areas of scientific

psychology.

8
The Uniform Guidelines on Employee Selection Procedures, published in 1978 (10),

apply to public, as well as private employers. They include two major requirements:

the determination of adverse impact and the demonstration of job-relatedness. If an

employer's selection system shows adverse impact against one or more selected

groups, then the job-relatedness of this system must be demonstrated. The debates

surrounding the proposed Civil Rights Act of 1990 have focused on defining these

terms "adverse impact" and "job-relatedness." For now, however, "adverse impact"

can be understood as the amount or degree of differential effect that a selection

procedure has against a protected group, that puts the employer on the defensive

and initiates an investigation of the selection procedure. This is primarily a legal or

administrative matter, although psychologists are concerned about

representativeness of samples, statistical significance, and so forth.

Job-relatedness is the concern of psychologists. Over the past 25 years it has been

understood as meaning the demonstration of the validity of the selection procedures,

with criterion-related validity providing the most pe-' asive evidence. Showing a

significant statistical relationship between the scor, on a predictor test and the

scores on some criterion, typically training grades or supervisory ratings of job

performance, has been convincing in many cases. Content validity can be

acceptable in certain circumstances, such as using a typing test to select for clerical

occupations. The FAA's Screen for ATCS, described below, is based on both a

content validity strategy--the Screen is a miniature training and test situation of how

a controller learns the job--and a criterion-related strategy--Screen grades predict

field training performance. A third strategy is construct validity where some

cognitive ability (e.g., the construct of spatial orientation of objects) is posited as

necessary to the ATCS occupation. Construct validation strategies can be risky

legal defenses unless their empirical grounding is well-articulated.

A major contribution to the understanding of validation in personnel settings is the

concept of validity generalization, initially proposed in 1977 by Schmidt and Hunter

(11). These authors, and others following, collected many criterion-related studies

and used the study results as data in meta-analyses. Validity coefficients vary

widely from one study to the next, a phenomenon many scientists had interpreted as

supporting "situational specificity;" that is, the use of one predictor test cannot be

generalized from the situation where the criterion-related validity study was

performed. Schmidt and Hunter (7) showed that this variability in validity coefficients

might be due not to variability in situations, or employment settings, but to the

varying numbers of subjects in the studies, the differing reliabilities of criterion

measures, and other artifacts; that is, criterion-related validities do generalize to

other situations.

There have been two major outcomes of the validity generalization movement--and it

does have some of the characteristics of a movement. First, it has restored both

the faith in and reliance on paper-and-pencil tests of general mental abilities as

entry-level predictors of success in most jobs. While some argue that a test of

general mental ability is useful for any occupation, the historical distinctions among

test types (e.g., verbal, quantitative, spatial) reflect small but important differences in

9
predicting job success. Such generic paper-and-pencil tests are still a major

component of personnel selection. Further distinctions among test types and the

mental processes required to perform well on them, such as those being explored

by cognitive psychologists, are usually of much less interest because these

distinctions are not reflected in predictiveness. However, those responsible for the

selection of air traffic controllers may have more interest in this cognitive approach,

because of test practice effects especially prevalent in spatial ability tests. Fairness

issues as well as validity are paramount in this approach.

The other outcome of validity generalization has been to focus more on the criteria

that the tests have been used to predict. Most of the validity generalization research is based on only two classes of criteria: training success and supervisory

ratings. By the time the validity generalization research began in the late 1970s,

civil rights activists and some personnel psychologists were questioning these two

criterion categories. While training is important, so went the argument, it is not the

job. Training may require more cognitive processing, which paper-and-pencil tests

predict fairly well, than may the job. As for supervisory ratings, these are typically

generic, probably hodge-podge, and possibly biased. Why, argued these critics, can

we not validate our selection procedures against closer surrogates to actual job performance?

The use of job performance measures in criterion-related validity research is fraught

with conceptual and logistical problems of high order. On the conceptual side,

should we deal with "typical" performance, that which the employee does on a

regular basis and that might be measured by detailed supervisor ratings? Or should

we try to measure "maximum" performance, what the employee can do, by

developing specialized situations in which the desired performance can be

evaluated? What types of performance should we measure? Should we measure

performance with a test of knowledge, or a demonstration, or a simulation? How

useful are archival measures, such as absenteeism? Logistically, most employees

are far more scattered in time and place than are applicants for a job. Further, since job incumbents tend to specialize, how can a single, albeit complex, performance measure be fair to all?

The research on performance measurement is not as advanced as that on tests and

predictors, but during the last decade there has been progress. The major effort

has been a joint-military service effort sponsored by the U.S. Department of

Defense. Preliminary answers to the questions raised above are (3) job

performance can be measured; (2) job performance measures can be used in

criterion-related validity research and can be predicted by multiple abilities test

batteries; (3) the most appropriate performance measures will differ from job to job,

e.g., knowledge tests would be appropriate for claims examiners but not for

mechanics; (4) all available performance measures, including archival measures can

be used to describe a "performance space," to provide a more complete picture of

the occupation and its requirements.

10
An emphasis on performance assessment, or on criterion-referencing, has been

growing steadily during the last two decades in many areas of psychological

research. The most prominent example is that of educational assessment. It no

longer is sufficient to have a comparison to "who," the traditional basis of normsreferencing; it must be a comparison to "what," or criterion-referencing, and why that "what" is important. Competency requirements for high school graduation are an

example.

Performance measures may also be used in ways not previously considered for

training grades and ratings, and it is this potential that is most exciting. For

example, let us say a work sample test were developed for a subgroup of Full

Performance Level (FPL) air traffic controllers. Say that this measure could be

judged as a representative sample of controller performance by an appropriate

sample of controllers and their supervisors, and that the test scores could be scaled

into ranges such as Outstanding, Exceptional, Fully Satisfactory, Marginal, and

Unacceptable. If this performance measure has a valid predictor that we would like

to use in our selection procedures, we can link the score levels on the work sample

to score levels on the predictor in both a more rational and more empirical manner

than is the current practice.

Thus, the state of the art in selection research is: (2) It is strictly regulated, very

public, heavily documented, and of mostly high quality; (2) Using paper-and-pencil

tests of general abilities has much technical justification; and (3) Performance

measures are becoming much more necessary in validation of selection procedures.

It is likely that computers will play an increasing role in selection research and in

selection, as they have in other areas of human endeavor. However, the question

for the technology will be the technical adequacy of the selection research program in which it is embedded. Research in selection of air traffic controllers of the future

will take place in this environment and must meet the existing standards and

regulations for employee selection.

SELECTION OF AIR TRAFFIC CONTROLLERS

History of the development of ATCS selection procedures.

The FAA has chosen to utilize a selection system prior to investing in training

individuals as ATCSs. The reason for the decision was a high attrition rate from the

occupation when no formal selection system was utilized prior to entry into training. The attrition was generally due to training failure and occurred, on the average, two

to three years into a person's career in air traffic (12). Such late attrition resulted in

considerable costs to both the FAA and the ATCS, who found little opportunity to

apply the ATC skills in other occupations. Toward the goal of reducing attrition as

well as identifying the individual characteristics required for success in the

occupation, the FAA developed an active selection research program.

11
As described by Brokaw (13), research on ATCS selection dates back to the 1950s

and his contract with the Civil Aeronautics Administration (CAA--precursor to the

FAA) for the development of a paper-and-pencil aptitude test battery that could be

used to select ATCS trainees. The results of the study indicated that some aptitude

tests reflecting content areas identified from a job analysis could potentially

contribute to the selection process. (This study provided the format for an Air

Traffic Problems test). In a joint effort with the CAA, the United States Air Force

Personnel Laboratory administered 37 aptitude tests to ATCS trainees in 1956. The

findings indicated that a composite aptitude test score--created by adding scores on

tests of arithmetic reasoning, symbolic reasoning, code translation, and the ATP

test--effectively predicted instructor's ratings of training performance and supervisors'

ratings of job performance approximately one year after training.

Research continued in 1960 with the establishment of the FAA Civil Aeromedical

Research Institute (presently the Civil Aeromedical Institute--CAMI), as described by

Collins, Boone, and VanDeventer (14). CAMI research on Civil Service Commission

(CSC) tests of similar content to the tests originally studied by Brokaw led the CSC

for the first time to establish tests for selecting ATCS trainees. Test requirements

for selection were implemented in 1962. The initial selection test battery consisted

of the following: Arithmetic Reasoning, Spatial Relations, Following Oral Directions,

Abstract Reasoning, and Air Traffic Problems. During the period between 1962 and

1972, in addition to continuing validation research, aviation psychologists studied a

number of factors relevant to ATCS selection: attrition from the profession, age,

prior experience, education, sex, and military ATC training.

Implementation of the CSC test battery alone failed to sufficiently decrease the

attrition rate. Prior to 1976, attrition rates from field training were 38%. A number

of studies were completed in the early 1970s, as well as Congressional hearings of

the Committee on Government Operations, which resulted in several

recommendations. As a result of the reports, the FAA developed a standardized,

centralized, validated program designed to identify, as early as possible, and remove

from training those candidates demonstrating insufficient aptitude to become ATCSs.

The program was designed to decrease the costs of attrition and improve the

selection of ATCSs. The FAA's aviation psychologists collaborated to provide a

scientific framework required for the design, implementation, and evaluation of the

program.

The result was the implementation of a second-stage selection procedure designed

to follow the CSC test battery. This second-stage hurdle consisted of two optionspecific (en route and terminal) pass/fail programs utilizing courses similar to those

used in the nonradar separation portions of the FAA Academy ATC training

curriculum existing at that time. Grades in these courses had been found to be

predictive of future field attrition (12). The programs were implemented in 1976.

During the latter half of the 1970s a new initial selection test was developed.

Research focused on studies relevant to the Uniform Guidelines on Employee

Selection Procedures, and on developing field training performance ratings,

12
developing a longitudinal database for continuing validation research, and refining

optimal combinations of old and new aptitude screening measures (14).

The current ATCS selection system.

The current ATCS selection system consists of two hurdles or stages of

assessment. The following sections present these procedures in more detail and

give statistics on each method's ability to predict other measures of ATCS

performance.

The OPM Selection Battery. The first stage of the selection process is the Office of

Personnet Management Air Traffic Control selection test battery. This stage is a

paper-and-pencil format aptitude test battery consisting of three tests: the Multiplex

Controller Aptitude Test (MCAT), Abstract Reasoning (included from the prior

selection battery), and the Occupational Knowledge Test (OKT). It provides the

usual benefits of paper-and-pencil tests, allowing initial assessment of large numbers

of potential candidates at relatively low cost.

The MCAT was developed to establish a paper-and-pencil test with higher predictive

validity than tests included in the CSC selection battery. The MCAT was designed

to approximate simulated air traffic activities. Simulated airspaces depict several

aircraft traversing the space. An altitude, speed and route chart is also presented.

Candidates are asked to identify situations that may result in conflicts between

aircraft based upon a limited set of separation criteria. Other items require

computing time-distance functions, interpreting information, and recognizing spatial

relations. The test is in a timed, multiple-choice format. The Abstract Reasoning

measures two principles of logical development exhibited by sequences of figures

and letters (14). The Occupational Knowledge Test (OKT) provides a more

objective and reliable measure of a candidate's ATCS job knowledge than that which

was provided by ratings based upon job history. Items on the OKT cover seven

knowledge areas related to air traffic control.

This battery, as is true of many others, including spatial ability tests, is highly

subject to practice effects. Generally, a second, higher score has lower validity than

the first score. To compensate for practice effects, applicants earning a score of 70

or above (out of a possible 100) presently cannot retake the OPM Selection Battery

for 18 months.

The FAA Academy Screen. The second stage of the selection process is the FAA

Academy Screen, a nine-week assessment program equivalent to a miniaturized

training-testing environment. Candidates are taught rules for aircraft separation and

13
are assessed in laboratory simulation problems on their ability to learn and apply the

rules. This pass/fail stage is very resource-intensive, but selects candidates likely to

succeed to full performance level from among those who have passed the initial

OPM battery.

The goal of the second stage program, which was implemented in 1976, was to

reduce field attrition through early identification of candidates with little chance of

succeeding in field training. Because the Screen is a miniaturized training-testing

assessment program, its purpose is to teach a candidate, with no knowledge of air

traffic control, enough about the job to assess the potential to succeed to the full

performance level. This is accomplished by training the candidate in aspects of the

job and assessing performance on a sample task (i.e., separating aircraft in

laboratory nonradar simulation problems). The complexities of the laboratory

problems are structured to escalate quickly, generally beyond most candidates'

ability to master the complexity levels being tested. This allows for a certain level of

discrimination of the better candidates from the candidates with less ability.

The program includes four weeks of didactic, classroom training on nonradar ATC

rules and principles. Content includes rules of separation (vertical, longitudinal, and

lateral), cooperative agreements, protection of special-use airspace, phraseology,

and flight progress strip-marking. Scme practice in applying the rules is provided in

the classroom. For the final five weeks, training focuses on application of the rules

in a nonradar laboratory. Each student performs nonradar controller job tasks in

one or two thirty-minute scripted scenarios each day. During each scenario an

individual instructor is assigned to each student. Following the scenario, the

instructor reviews the student's performance with the student, providing feedback on

errors, correct application of rules, and other required skills.

Thirteen performance measures are made of each candidate during the nine-week

period. Five of these assessments are multiple-choice tests on the academic

portion of the Screen. These assess the candidate's ability to learn and retain the

basic knowledge required for the job. A sixth measure is a map test which

assesses the students' ability to learn a map of the relatively simple synthetic

airspace. These measures account for 20% of the final grade. Six of the

performance measures are formal evaluations of the student's performance during

2ix 30-minute standardized scenarios. The score on these laboratory simulation

problems consists of a technical assessment (number of errors) and an instructor

assessment (subjective evaluation of student performance). Only five of the six

laboratory problems are included in calculating the final score. These five scores,

however, comprise 60% of the final grade. Finally, the students are given a timed,

multiple-choice test, the controller skills test (CST), which is a paper-and-pencil

assessment of the students' ability to apply ATC rules and procedures. The CST

comprises 20% of the final composite grade. A final grade of at least 70 (out of a

possible score of 100) is required to pass the Screen. Failure to achieve a

composite score of 70 results in removal from the ATCS job.

14
The ATCS selection procedures and field training status and attrition rates.

As noted, the Academy screening programs were implemented in 1976 to provide

early identification and removal of candidates with little chance of success in the

occupation and thus reduce field attrition rates. Prior to 1976, the attrition rate in

field training was 38%. After implementation of the screening programs, attrition

rates at the Academy averaged 30% and field training attrition declined to 8% (15). Following the 1981 controllers' strike and subsequent increase in the population of

Academy entrants with no prior ATC experience, the attrition rate at the Academy increased to 40%, while the field attrition rate increased to 11%. Thus, the

Academy screening programs were effective in decreasing attrition from field

training.

To determine the extent to which the original OPM and Screen measures predicted

subsequent measures of performance in field training in the en route (ARTCC) environment, Manning, et al, (16) compared correlations between the OPM ATCS

selection test scores and scores from the FAA Academy Screen with measures of

field training performance for competitive Academy students who entered between

September 1981 and September 1985. Of the OPM measures, the MCAT was

found to be a better predictor of field training status in the en route option than

other OPM tests. Academy laboratory performance scores were found to be more

predictive of field training performance than the academic test scores. Finally, for

the en route option, Academy performance measures, particularly laboratory scores,

were better predictors of supervisor and on-the-job training (OJT) instructor ratings and training status than were OPM scores.

Current correlations between performance on the OPM battery, Academy Screen,

and field training status.

In October 1985, a change was made to the Academy Screen which resulted in

reweighting the performance measures. Because the average time for an en route

ATCS to attain full performance level (FPL) status is just under 3 years, we are just

now able to evaluate the impact of this change on predictive validity. The analyses reported here are an initial evaluation of the change. Because this paper has

primarily focused on how increasing automation affects the en route controller, the

analyses address only the en route option. In addition, a modification to the OPM

test battery administration procedures was also implemented in October 1985.

Because the affect of the change has not yet been evaluated, this paper presents

only data from ATCSs who took the version of the OPM test prior to the

modification.

To assess the validity of the new Academy Screen, analyses were conducted to

examine the distributions of OPM and Academy scores for first-time competitive

15
students who entered the Academy between October 1985 and September 1986,

(i.e., during FY-1986). As noted, the sample included only students with OPM

ratings from the former administration procedures and only those students assigned

to en route facilities. Finally, ATCS developmentals were excluded from the sample

if they withdrew from the career field for reasons other than failing training (e.g., to

pursue another career).

Appendices B-E show statistics related to the distributions of OPM test scores and

Academy performance measures. Means and standard deviations of OPM test

scores for all applicants who took the OPM test between April and October 1985

are shown in Appendix B. Means and standard deviations of OPM test scores for

those who entered the Academy between October 1985 and September 1986 and

took the prior administration of the OPM test are shown in Appendix C. Means and

standard deviations for Academy performance measures are shown in Appendix D.

Appendix E shows correlations between OPM test scores and Academy performance

measures.

Tables 1 and 2 show correlations of OPM and Academy performance measures with

a variable measuring status in field training, both unadjusted and adjusted for

restriction in the range of predictor scores, respectively. The variable "field training

status" has the following categories: 1) reached FPL, 2) still in training, 3) switched

options, and 4) failed. The tables compare the correlations reported in the Manning,

et a, (16) study with those computed for the FY-1986 sample of Academy Screen

graduates assigned to the en route option. It can be seen that the correlations are

similar. Examination of Table 2 shows that the correlation between the OPM rating

and field training status, corrected for restriction in the range of OPM rating, is

higher than was found for the 1981-1985 sample, although the correlations of

individual OPM component scores with the criterion were similar.

16
TABLE 1

CORRELATIONS BETWEEN OPM AND ACADEMY SCREEN PERFORMANCE

MEASURES

AND MEASURES OF FIELD TRAINING STATUS

For 1981-1985 En Route and 1986 Screen Entrants

(Uncorrected for restriction in range)

1981-1985 1986

En Route Screen

(n=2992) (n=402)

OPM TEST MEASURES

Multiplex Controller

Aptitude Test (MCAT) -.12* -.09

Abstract Reasoning .03 -.03

Occupational Knowledge

Test (OKT) .00 -.05

Transmuted Composite -.10* -.08

(TMC)

OPM Rating -.05* -.09

ACADEMY SCREEN MEASURES

Block Test Average -.05" -.06

Controller Course Test -.04 -.07

Ave. 5 of 6 Laboratory

Problems -.24* -.21

Ave. Instructor

Assessment -.25* -.22*

Ave. Technical

Assessment -.22* -.21

Controller Skills

Test -.08* -.16*

Comprehensive Score -.24* -.24*

•Correlations were significantly different from 0 at p<.01.

17
TABLE 2

CORRELATIONS BETWEEN OPM AND SCREEN PERFORMANCE MEASURES

AND MEASURES OF FIELD TRAINING STATUS

For 1981-1985 En Route and 1986 Screen Graduates

(Adjusted for restriction in the range of predictor scores)

1981-1985 1986

En Route Screen

(n=2992) (n=402)

OPM TEST MEASURES

Multiplex Controller

Aptitude Test -.28 -.24

Abstract Reasoning .05 -.04

Occupational Knowledge

Test .00 -.04

Transmuted Composite -.25 -.24

OPM Rating -.13 -.30

ACADEMY SCREEN MEASURES

Block Test Average -.09 -. 10

Controller Course

Test -.05 -.09

Ave. 5 of 6 Laboratory

Problems -.42 -.36

Ave. Instructor

Assessment -.46 -.37

Ave. Technical

Assessment -.33 -.30

Controller Skills Test -.14 -.26

Comprehensive Score -.46 -.44

Also, the correlations of the Laboratory Average (average score on best 5 of 6

laboratory problems) and Laboratory Instructor Assessment (average score on

instructor subjective rating in laboratory problems) with en route field training status

decreased somewhat, while the correlation between the CST (Controller Skills Test) and en route field training status increased somewhat. For the remaining measures,

the similarity in correlation coefficients for the two samples suggests that the validity of the program in predicting status in field training has remained stable, in spite of

the changes made to the Academy Screen.

18
Use of Nonradar ATC Tasks to Predict a Radar-based Job

The Screen program is based upon nonradar air traffic separation rules. At the time

it was developed, nonradar procedures were used more frequently. Today, radar

separation procedures are utilized almost exclusively in the majority of the ARTCC's.

Thus, the question arises about the extent to which a program that is based on

nonradar procedures may effectively predict success in a radar environment.

A comparison of the ATC tasks trained in the Academy Screen to the tasks required

of an en route controller suggests that many of the behaviors measured in the

current Screen are similar to those required in the radar environment. Of the six

activities of an en route controller identified in the CTA job task analysis (1. perform

siti'ation monitoring, 2. resolve aircraft conflicts, 3. manage air traffic sequences, 4.

route or plan flights, 5. assess weather impact, and 6. manage sector/position

resources), the Screen assesses tasks associated with five of the six activities.

Only weather assessment (activity 5) is not taught or tested in the Screen program.

With the exception of the tasks that require checking the plan view display (PVD) or

utilizing radar separation procedures, most of the tasks in the composition graph are

similar to the nonradar training taught and evaluated in the Screen. For example,

the first sub-activity under "perform situation monitoring" is checking and evaluating

the separation of aircraft. The sequence of tasks for this sub-activity proceeds as

follows: i. Review of flight progress for present and/or future aircraft separation; ii.

Review of PVD (or flight progress strips in nonradar) for potential violation of

airspace separation standards; iii. Review of location of flight progress strip in bay, if

unsatisfactory, resequence manually; iv. Mental assessment of potential conflict--

project mentally an aircraft's future position/altitude/path and evaluate mental flight

plan projection for appropriateness; v. Range/bearing data may assist assessment of

possible conflict--request range/bearing/time (by contacting the pilot in nonradar). Of

18 tasks described for the sub-activity, 12 tasks or nonradar correlates are assessed

in the Screen. This is an example of the extent to which the tasks taught and

assessed in the Screen can be related to the tasks required of an en route

controller even though radar equipment has added an automated dimension and

new, specific air traffic separation procedures.

Knowing that performance on nonradar tasks has some utility in predicting future

success in the more highly-automated radar-based occupation may help develop

selection procedures for the automated systems of the future. Perhaps selection

procedures designed for the current occupation will predict performance in the

evolving occupation.

Because of the ATCS strike in 1981, development of a radar-based selection

program was delayed. Recently, however, interest and enthusiasm for addressing

questions of new and improved selection procedures for ATCSs have been revived.

19
COMPUTERIZED SELECTION SYSTEMS

Since the development of the FAA's selection procedures, the use of computers in

assessment has become more feasible and may have some utility for future

selection of ATCSs. Much interest and effort during the last two decades have

been devoted to creating computer-based assessment and selection systems. A

number of computer-based assessment systems are available commercially. Some

are directed towards clerical occupations while others are marketed for both

vocational guidance and selection, using computer-adaptive multiple cognitive

abilities batteries.

Further selection applications have been few. However, several problems

complicate the application of computerized testing to the selection of employees.

First, typical test construction problems of standardization also apply to computerized

tests, as well as the computer-specific problems of calibration and user-friendliness.

Second, when an organization has computers, it is not hard to justify the use of

some of them for testing. When the selection function requires its own set of

computers, it is difficult for the organization to justify the use of all of them in

testing. Thus, large-scale computerized selection programs, such as might be

expected in the military services, have been difficult to implement. Third, the

selection research base for abilities measurable only by computer, such as dynamic

spatial ability and processing speed, remains comparatively slirr. The three military

research laboratories in the United States have each developed computerized

assessment systems, which measure abilities different from those assessed by

paper-and-pencil selection tests. As this research base develops, computers are

likely to play a larger role in selection.

FUTURE SELECTION SYSTEMS

This paper has presented an overview of the NAS Plan for automation of the air

traffic controllers' job over the next two decades. To the extent that changes in job

tasks have been projected so far, we have attempted to map them for the Initial

Sector Suite System and Automated En Route Air Traffic Control. At this time, it

appears that the job of the ISSS controller will require many of the same skills and

cognitive attributes required today. The implementation of AERA, however, should

begin to substantially change the involvement of the controller from a tactical mode

of operation to a more strategic mode.

Clearly, much of the controller's activity when operating AERA 2 will involve

examining data, testing resolutions, and communicating with the computer, aircraft,

or other controllers. While a controller will still need to be able to visualize

interrelationships between aircraft, the need to make rapid decisions using the

information will be reduced. The description of controller activities provided by

MITRE suggests that it might be appropriate to select controllers with different types

of skills, emphasizing strategic instead of tactical planning. This would suggest that

20
a different battery might be appropriate for selecting AERA 2 ATCSs.

The problem with the description of the future controller being exclusively one who

executes the computer's suggestions is that different skills may be required if control

actions are required in a short-term time frame, perhaps a shorter time frame than

the computer can resolve, or during emergencies. For example, if events (e.g., VFR

popups) occur that are not identified by the computer, then the controller may be

required to respond quickly to avoid loss of separation between aircraft. These

actions are complicated by the proposed "direct routing" of aircraft from one

destination to another, without using designated airways. If emergencies requiring

controllers to engage in tactical control activities occur only rarely, and if tactical

ATC is be complicated by direct routing, then perhaps the future controller will

require skills at separating traffic that exceed those required today. As long as the

controller retains responsibility for the movement of all aircraft in the assigned

airspace (even though the computer ordinarily provides resolutions to potential conflicts), then he or she will have to possess and be able to maintain the KSAs to "manually" separate those aircraft, in case of emergency. Therefore, in this case,

the selection of ATCSs must be based on the requirement to perform under the

worst-case scenario. If, over time, the AERA software is accepted as accurate, and

is found to be able to handle all emergency situations, then the responsibility for the

aircraft may be removed from the ATCS and assigned to the computer, and the

controllers' job requirements may be changed, resulting in modifications to selection

procedures.

As plans for AERA 3 becom0 r,.)ie fully developed, the tradeoff of separation responsibilities between the ATUS and the computer will become more pronounced.

Preliminary projections arq that the AERA 3 software will undertake all control

activities, perhaps leaving the ATU6 Ic S-i c.learances to aircraft that do not have

data link capabilities, or perform traffic management activities. The controller may

also be responsible for ensuring that the computer is operating properly. In all

cases, even through AERA 3, the computer is dependent on timely, accurate input

of information from human operators and is, thus, vulnerable to the reliability and

accuracy of the human. Again, if the controller has responsibility for taking over the

control of air traffic when (if) the computer fails, then the selection procedures must

identify those who have the ability to control traffic, even if they only do so on rare

occasions. If, on the other hand, the full responsibility for aircraft separation lies

with the computer (perhaps yet another level of software), then selection procedures

can be targeted to the KSAs required to perform the duties most frequently

performed by the ATCS using AERA 3.

Development of any selection procedure depends on an accurate description of the

lasks to be performed and the knowledge, skills, and abilities (KSAs) required t"o,

perform those tasks. The problem with applying this approach to the development

of selection procedures for the "automated controller" is obtaining a sufficiently

detailed job/task analysis prior to implementation of the new system, because no

controllers will have used the system extensively enough to qualify as subject matter

experts (SME's). A fallback position would be to work with controller members of

21
the Air Traffic AERA Concepts Team (ATACT), who are as knowledgeable as any

controllers about the way the new system will work. These "near" SMEs could

speculate about the requirements to perform the tasks. Preliminary work can be

done early in the development cycle to identify or develop tests to measure the

KSAs, as they are currently identified. Review cycles will have to be built into the

process to allow periodic review of the KSAs with SMEs as more controllers are

brought into the loop to test the AERA software and procedures, and as the

software and procedures become increasingly well-defined. Using such an iterative

process, a tentative set of selection procedures would be available upon

implementation of the new system.

When contcmpiating the development of selection procedures fo.- employees who

will be ATCSs under the AERA levels of automation, it is clear that work must begin

now for the procedures to be ready when AERA 2 software is installed. At the

same time, the division of duties between the software and the controller have not

yet been finalized, so the final choice of a selection procedure is premature.

Because of the criticality of the ATCS occupation, it may not be possible to select

everyone who applies for the job and passes the new selection procedure for a

period of time until the validity of the new selection battery can be established.

When converting to the AERA system, it may be necessary to also continue using

the old selection procedures for awhile, until the validity of the new procedures can

be evaluated. The correlations between measures of performance in the nonradar

Screen and success in radar-based field training suggest that using a selection

procedure current, for example, for the radar job to select controllers for the new

ATC job under ISSS may be an acceptable interim solution while validation of the

new procedures occurs.

Furthermore, several years may be required to establish the validity of new selection

procedures. Most of that time is needed for those selected to complete field training

so that on-the-job performance can be measured, and the relationships between

selection predictors and job performance criteria can be assessed. Measures of

performance in training can be used as interim criteria, but, as previously mentioned,

are less desirable than measures of job performance.

DISCUSSION

The research from the past 30 years has produced some findings that can

contribute to the development of selection procedures for automated systems. First,

paper-and-pencil aptitude tests alone have been found insufficient to predict

performance on the complex procedural tasks involved in ATC in its current

environment. Second, the addition of a second-stage miniaturized training-testing

program has improved the utility and validity of the ATCS selection process and

reduced field attrition. This selection program, based upon nonradar separation, has

been useful for predicting success in the radar environment.

22
As with selection research in other occupations, the research on air traffic control

specialists suffers from lack of availability of criterion measures of ATC's job

performance. Because the Screen measures task samples via the laboratory

problems, the experience gained in developing measures of training performance

for both the Screen and the Academy basic radar training courses could contribute

to future efforts to develop criterion job performance measures for the automated

ATC environment.

The current selection system has been found to have utility in decreasing the field

attrition rate; however, it may not be the best selection system for radar ATCSs or

the evolving automated occupation, and it is certainly very costly in terms of

monetary and human resources. Therefore, the FAA is beginning to explore

alternatives to the current methods. The goals are to have a shorter and less costly

selection process, with predictive validity similar to the present system, in place

within the next five years.

Clearly, the development of selection procedures for automated systems will be a

difficult process, complicated by a lack of information about the job, varying

expectations about eventual job task requirements, extensive time frames for

validation, and a need to develop appropriate criterion measures of job performance.

The time to start addressing these issue is now, so that development of selection

procedures for the changing occupation can parallel the evolution of the job.

23
REFERENCES

1. Hopkin DV. Training implications of technological advances in Ai. Traffic

Control. In Proceedings of Symposium on Air Traffic Control training

for tomorrow's technology, Oklahoma City, OK, December 6-7, 1988.

2. U. S. Department of Transportation, Fpderal Aviation Administration. National

Airspace System Plan: Facilities, Equipment, Associated Development

and Other Capital Needs, 1989.

3. Gutmann J. Advanced Automation System, Human computer interface: An

overview. In Proceedings of Symposium on Air Traffic Control training

for tomorrow's technology, Oklahoma City, OK, December 6-7, 1988.

4. Ammerman HL, Becker ES, Jones GW, Tobey WK, and Phillips MK. FAA Air

Traffic Control Operations Concepts Volume k A TC Background and

Analysis Methodology. Washington D.C.: Federal Aviation

Administration Report No. DOT/FAA/AP-87-01, 1987.

5. Ammerman HL, Bergen LJ, Davies DK, Hostetler CM, Inman EE, and Jones

GW. FAA Air Traffic Control Operations Concepts Volume VI:

ARTCC/HOST En Route Controllers. Washington, D.C.: Federal

Aviation Administration Report No. DOT/FAA/AP-87-01, 1987.

6. Ammerman HL and Jones GW. ISSS impact on ATC procedures and

training. Colorado Springs, CO: CTA Incorporated. Contract DTFA0i-85-Y-01034, CDRL C108, 1988.

7. Alexander JR, Alley VL, Ammerman HL, Hostetler CM, and Jones GW. FAA

Air Traffic Control Operations Concepts Volume Ill: ISSS En Route

Controllers. Washington D.C.: Federal Aviation Administration Report

No. DOT/FAA/AP-87-01, 1988.

8. Carlson LS, and Rhodes LR. Changes in Controllers' work activities in

AERA 2 and implications for controller training. MITRE technical report

No. MP-89W00021, January 1990.

9. McKinley J, and Jago R. AERA En Route Controller Skills Requirements.

Unpublished manuscript, September, 1984.

10. Uniform Guidelines on Employee Selection Procedures, Federal Register,

1978, 43, 38290-83815.

11. Schmidt FL, and Hunter JE. Development of a qeneral solution to the

problem of validity generalization. Journal of Applied Psychology, 1977,

62, 529-540.

24
12. Cobb BB Jr., Mathers JJ, and Nelson PL. Attrition-Retention rates of air

traffic control trainees recruited during 1960-1963 and 1968-1970.

Washington, D.C.: Federal Aviation Administration Office of Aviation

Medicine Report No. FAA-AM-72-33, 1972.

13. Brokaw LD Early research on controller selection, 1941-1963. In Sells SB,

Dailey JT, and Pickrel EW. (Eds.), Selection of air traffic controllers.

Washington, D.C.: Federal Aviation Administration Office of Aviation

Medicine Report No. FAA-AM-84-2, 1984.

14. Collins WE, Boone JO, and VanDeventer AD. (Eds.), The selection of air

traffic control specialists: History and review of contributions by the

Civil Aeromedical Institute, 1960-1980. Aviat. Space Environ. Med.

1981, 52, 217-240.

15. Manning CA, Kegg PS, and Collins WE. Selection and screening programs

for air traffic control. In Jensen, R. S. (Ed.) Aviation Psychology.

Aldershot, England: Gower Technical, 1988.

16. Manning CA, Della Rocco PS, and Bryant KD. Prediction of success in FAA

air traffic control field training as a function of selection and screening

test performance. Washington, D.C.: Federal Aviation Administration

Office of Aviation Medicine Report No. DOT/FAA/AM-89/6, 1989.

25
APPENDIX A

Cognitive/Sensory Attribute Definitions from Computer Technology Associates' Job

Task Analysis (4).

CODING Transformation or translation of

information for entry into the system;

Converting textual information to

graphics or symbols.

Example: Entering a PIREP (pilot report); Composing a

flight plan amendment.

DECODING Transformation or translation of

information received.

Example: Recognizing a symbol for a

handoff; Reading a Flight Data Entry.

DEDUCTIVE Ability to reach a conclusion that

REASONING follows logically from the known facts

or data; Selection from among

alternative answers or methods.

Example: Concluding that two aircraft

are on intersecting paths

FILTERING Selection of inputs on which to focus

attention in the presence of distracting

stimuli or high workload; Selective

attention; Overload accommodation.

Example: Identifying communication transmissions

for attention during a period of heavy radio traffic.

IMAGE/ Perception of spatial patterns and

PATTERN relations among static or dynamic

RECOGNITION visual inputs. May involve orienting

oneself to the picture or configuration.

Example: Forming a picture of the traffic situation

by reviewing Flight Data Entries on the Flight

Data Display.

INDUCTIVE Generation of an explanation for a set

REASONING of specific data or instances, giving

Al
structure and meaning to the information;

Generalization of working hypotheses from

specific events;

INDUCTIVE Discerning basic differences and

REASONING (cont.) relationships among symbols, figures

and figure patterns; Generating a new

solution to a problem; Ability to make

a knowledgeable assumption using incomplete

data.

Example: Checking the adequacy of a proposed aircraft

maneuver.

LONG-TERM Mental storage of knowledge over a

MEMORY period of time and selective recall

of items relevant to a situation.

Example: Remembering aircraft characteristics; Remembering

procedural instructions or letters of agreement

relevant to an uncommon situation, such

as an airshow or large flight formation.

MATHEMATICAL/ Translation of uncertainty into

PROBABILISTIC probability; Assigning a subjective

REASONING probability regarding the likelihood of

an event occurring; Ability to use

probabilities to identify optimal

courses of action.

Example: Assessing the risk of an aircraft maneuver.

MOVEMENT Recognition of the physical movement

DETECTION of a visual object; Estimation of its

direction or speed.

Example: Observing aircraft on the Situation Display responding

to a clearance or advisory.

PRIORITIZING Ordering of events in sequence;

Establishing priorities.

Example: Deferring a request for flight plan changes in the

presence of more urgent activity.

SHORT-TERM Mental storage and selective recall of

MEMORY of relevant information over a brief

period of time.
Example: Briefly retaining and entering

an aircraft call sign.

SPATIAL Rapid identification or detection of

SCANNING objects or events displayed in a wide

or complicated visual field.

Example: Observing the Situation Display for new aircraft;

Searching for data in a table.

VERBAL Same as FILTERING, but limited to voice

FILTERING communications.

VISUALIZATION Observation of spatial patterns and

subsequent mental transformations into

other spatial patterns.

Example: Determining the effect of a proposed aircraft

maneuver on other aircraft; Comparing intended

time-position profiles for intersection in

position/altitude/time.
APPENDIX B

Applicant OPM ATCS battery performance

Mean Scores on OPM ATCS battery component tests for

OPM open period from April-October 1985

N=8,826

Measure Mean Std. Dev.

Multiplex Controller

Aptitude Test (MCAT) 73.0 16.6

Abstract Reasoning (ABSR) 31.5 9.3

Occupational Knowledge

Test (OKT) 29.2 11.6

Transmuted Composite 75.5 12.5

(TMC)

OPM Rating (RAT) 76.4 13.3

BI
APPENDIX C

Academy Entrant OPM ATCS battery performance

Mean Scores on OPM ATCS battery component tests

for October 1985 through September 1986 Academy Entrants

N=402

Measure Mean Std. Dev.

Multiplex Controller

Aptitude Test 94.3 5.8

Abstract Reasoning 40.6 5.5

Occupational Knowledge

Test 36.5 14.6

Transmuted Composite 91.8 4.3

OPM Rating 93.7 4.0

C1
APPENDIX D

Academy Performance Measures

Mean Scores for Academy Performance Measures

October 1985 through September 1986 Competitive Entrants

N=402

Block Test Ave. 95.7 4.4

Controller Course

Test 93.4 5.4

Ave. 5 of 6 Laboratory

Problems 74.0 7.1

Ave. Instructor

Assessment 84.6 5.5

Ave. Technical

Assessment 54.1 10.8

Controller Skills

Test 82.0 7.5

Comprehensive Score 79.7 5.0

DI
APPENDIX E

Correlations between Academy and OPM battery component scores

FY 1986 competitive entrants assigned to en route option

N=402

OPM measures

Measure MCAT ABSR OKT TMC RAT

Academy measures

Block Test Ave. .02 .03 .14* .02 .11

Controller Course

Test .01 .02 .25* .02 .19*

Ave. 5 of 6 Lab.

Problems .22* .04 .00 .20* .15*

Ave. Instructor

Assessment .26* .05 .03 .25* .19*

Ave. Technical

Assessment .23* .05 .01 .22* .17*

Controller Skills

Test .16* .12* .03 .19* .18*

Comprehensive

Score .24* .08 .05 .23 .21*

*Correlations were significantly different from 0 at p<.01.

('.R , MFT PRrNTfNG OFEICF: 1990--,61-O20/20 4 4 0

El