Air Traffic Scenarios Test: 

Will Modifying the Instructions 

Change Performance?

Alfretia L. Scarborough 

M. Katherine Bleckley

Civil Aerospace Medical Institute

Federal Aviation Administration

Oklahoma City, OK 73125

April 2015

Final Report

DOT/FAA/AM-15/7

Office of Aerospace Medicine

Washington, DC 20591

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship 

of the U.S. Department of Transportation in the interest 

of information exchange. The United States Government 

assumes no liability for the contents thereof.

___________

This publication and all Office of Aerospace Medicine 

technical reports are available in full-text from the 

Federal Aviation Administration website.
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-15/7

4. Title and Subtitle 5. Report Date

Air Traffic Scenarios Test: Will Modifying the Instructions 

Change Performance?

April 2015

6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Scarborough AL, Bleckley MK

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

FAA Civil Aerospace Medical Institute

P.O. Box 25082

Oklahoma City, OK 73125

11. Contract or Grant No.

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aerospace Medicine

Federal Aviation Administration

800 Independence Ave., S.W.

Washington, DC 20591 14. Sponsoring Agency Code

15. Supplemental Notes

Work was accomplished under approved task AM-B-HRR-523

16. Abstract

The Air Traffic Scenarios Test (ATST) is a low-fidelity radar simulation that is part of the Air Traffic 

Training and Test (AT-SAT) battery. Past Federal Aviation Administration (FAA) and military research has 

shown the ATST to be a strong predictor of training success for air traffic control specialists (ATCSs). 

Because the ATST was such a strong predictor of controller performance, the U.S. Air Force (USAF) was 

interested in using the ATST as part of its entrance testing selection process for air traffic. However, the 

current version of the ATST takes about an hour and a half to complete, and is too long for the USAF to use 

in its present form. In agreement with the FAA, the USAF requested that the FAA shorten (can be 

completed in less than an hour) the ATST without jeopardizing its reliability and validity. Two studies were 

conducted to examine whether it would be possible to shorten the ATST.

In the preliminary study (Study 1), we analyzed data from 391 job applicants that took the AT-SAT 

during a two-month period. We examined scores for efficiency of aircraft movement to destination, safety 

or maintenance of aircraft separation, and procedural accuracy. Overall, performance changed across trials.

Because performance did not consistently increase or decrease across trials, we cannot simply reduce the 

number of trials without changing the validity of the test. We thought it was possible that, with a more 

structured approach to designing instructions, performance on the trials may become more consistent, thus 

allowing us to reduce the number of trials to yield a reduction in total testing time.

For Study 2, new interactive instructions were developed. Data were collected from Air Force recruits 

who were naïve about the ATST. A final sample size of 193 cases was used in the data analysis. Participants 

spent 6 minutes longer on the modified instructions than on the standard ones; however, both times were 

substantially less than the currently allotted time of 25 minutes. ATST performance for both groups was 

poor. Although a profile analysis of the two instruction types showed no significant difference, the 

multivariate tests for trial revealed a change across trials for the three measures. Additionally, when we 

compared the results of this study with our original sample of ATCS applicants, we found large differences 

in performance that might indicate that the USAF recruits were physically tired and had less motivation to 

perform well than did our applicants; accordingly, their performance was poorer. In sum, the results of these 

studies were inconclusive with regard to whether a shorter ATST would be viable as a controller selection 

instrument. Additional research will be necessary to determine if the modified instructions are beneficial 

and whether fewer scenarios will maintain similar or improved predictive validity for ATST.

17. Key Words 18. Distribution Statement

Air Traffic Scenarios Test, Air Traffic Selection, 

Improving Test Instructions

Document is available to the public 

through the Internet:

www.faa.gov/go/oamtechreports

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 19

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

iii

ACKNOWLEDGMENTS

Research reported in this paper was conducted under the Air Traffic Program Directive / Level of Effort Agreement 

between the Human Factors Research and Engineering Group (AJP-61), FAA Headquarters and the Aerospace Human 

Factors Division (AAM-500) of the Civil Aerospace Medical Institute.

We wish to express our thanks and appreciation to the United States Air Force researchers at Lackland Air Force 

Base in San Antonio, Texas, for collaborating with us on this project and for the use of the Applied Performance 

Assessment and Testing (APTAT) facility.

We also express thanks and appreciation for the contributions to this report from each of the reviewers.

v

Contents

Air Traffic Scenarios Test: 

Will Modifying the Instructions Change Performance?

INTRODUCTION. 

STUDY 1: PRELIMINARY ANALYSIS. 

Method. 

Data. 

Testing Platform. 

Design and Data Analysis. 

Results. 

Discussion. 

STUDY 2: DEVELOPMENT AND EVALUATION OF MODIFIED TESTING INSTRUCTIONS. 

Method. 

Design and Data Analysis. 

Procedure. 

Results. 

Air Force Recruits. 

Performance Comparison of Air Force Recruits Versus FAA applicants. 

Discussion. 

Hypothesis 1. 

Hypothesis 2. 

Performance Comparison. 

REFERENCES. 

APPENDIX A. ATST Practice and Test Duration (in minutes). 

APPENDIX B. An Example of the ATST Standard Instructions. 

APPENDIX C. An Example of the ATST Modified Instructions. 

1

Air Traffic Scenarios T

Will Modifying the Instructions Change Pe

INTRODUCTION

The Federal Aviation Administration (FAA) has gone through 

various transitions in its selection program for Air Traffic Control Specialists (ATCSs). In 2000, the Air Traffic Selection and 

Training Battery (AT-SAT) was developed to be used as part 

of the ATCS screening process (Ramos, Manning, & Heil, 

2001a). The AT-SAT test battery contains eight subtests. One 

of the eight AT-SAT subtests is the Air Traffic Scenarios Test 

(ATST), a low-fidelity simulated radar control task designed 

to measure 11 air traffic controller abilities or worker requirements (Nickles, Bobko, Blair, Sands, & Tartak, 1995, see Table 

1). Past research has shown the ATST to be a strong predictor 

of training success for air traffic controllers (Carretta & King, 

2008;Broach & Brecht-Clark, 1993; Weltin, Broach, Goldbach, 

& O’Donnell, 1992).

The ATST is a dynamic low-fidelity simulation of the en route 

air traffic controller work environment. The objective of the test 

is to maintain separation of aircraft by following established rules. 

Abilities tested are: learning and following simple directions, 

visualizing and projecting paths in three dimensions, monitoring several objects at once, planning ahead, and remembering 

to execute simple commands at the appropriate time. Elements 

of the test include a bordered area representing the airspace, 

control panel, aircraft (represented as arrows), data blocks that 

show current speed, altitude, and heading; four exits; and two 

airports (see Figure 1). The mouse is used to issue commands 

(change speed, altitude, or heading). Audio feedback, representing a pilot’s voice, acknowledges the changes. 

Figure 1. Diagram representing the Air Traffic Scenarios Test screen.

Table 1

Worker Requirements for Air Traffic Scenarios Test

ATST Worker Requirements

Prioritization

Tolerance for high intensity

Composure

Situation awareness

Planning

Execution

Thinking ahead

Reasoning

Decisiveness

Scanning

Perceptual speed and accuracy
2

The test produced three subscores: an efficiency score, a safety 

score, and a procedural accuracy score. The efficiency score is 

based on the speed at which aircraft reach their destinations 

and how quickly aircraft are accepted as they enter the airspace. 

The safety score is based on the maintenance of separation of 

aircraft from each other and from the border of the airspace by 

an established distance. The procedural accuracy score is based 

on following the rules for accurately landing aircraft and exiting 

them from the airspace. Aircraft must land at the slowest speed 

(S) and lowest altitude (1) when heading toward an airport. 

Aircraft must exit the airspace at the fastest speed (F) and highest altitude (4). 

Carretta and King (2008) suggested that the U.S. Air Force 

(USAF) could benefit from adding the ATST as part of its selection process for air traffic controllers (ATCs). When they used 

ATST as an adjunct to the Armed Services Vocational Aptitude 

Battery (ASVAB), the selection battery for the USAF and other 

military branches, the results showed that the ATST significantly 

improved prediction of USAF trainees’ performance beyond the 

ASVAB (completely corrected ∆R2 =.156).

The current version of ATST is too long to be utilized at 

Military Entrance Processing Stations (MEPS); therefore, the 

USAF requested a version of ATST that could be completed 

in less than one hour. Because of this, we examined the ATST 

to determine if the test could be shortened while maintaining 

its reliability and validity. A shorter version of ATST would 

require less time to administer and therefore could be useful to 

the FAA, as well.

We conducted two studies to determine if it would be possible 

to shorten the ATST while retaining its value as a controller selection instrument. In Study 1, a preliminary analysis, we wanted 

to know if there was evidence of performance leveling off across 

testing trials in the current version of the ATST to determine 

if any trials could be dropped to shorten the ATST testing 

time. This will help us determine if the duration of the ATST 

can be reduced without jeopardizing its reliability and validity 

(Scarborough & Bleckley, 2008). In Study 2, we thought that 

using interactive instructions might reduce testing time through 

improved and more stable performance in the earlier trials. 

We set out to answer our research question: Is there evidence 

of performance leveling off across testing trials?

STUDY 1: PRELIMINARY ANALYSIS

Method

Data

Archived data collected in 2007 from 391 job applicants that 

took the ATST operationally during the two-month period , 

July and August, were analyzed.

Testing Platform

The ATST consists of three practice scenarios (2-1 minute 

scenarios, 1-3 minute scenario) and four test scenarios. The first 

three test scenarios, or trials, are 15 minutes long, increasing 

in number of aircraft with each trial (trial 1 = 18 aircraft, trial 

2 = 20 aircraft, and trial 3 =31 aircraft). The last test trial is 25 

minutes long and contains the most aircraft (45). Including 

the 25 minutes allotted for instructions, the test currently takes 

approximately 95 minutes to complete (Appendix A).

Design and Data Analysis

We examined applicants’ performance scores by trial for 

efficiency of aircraft movement to destination (efficiency), maintenance of aircraft separation (safety), and procedural accuracy. 

A multivariate repeated-measure ANOVA was conducted that 

used the ATST performance variables described above and trials 

as within-subject factors. 

Results

Results from the multivariate repeated measures ANOVA 

showed that the Wilk’s Lambda was significant, F (6, 385) = 

131.88, p = .00, for the effect of trials. Univariate ANOVAs

showed that the effect of trials varied across all three performance 

variables: efficiency, F (1, 390) = 72.67, p = .00; safety, F (1, 390) 

= 129.98, p = .00; and procedural accuracy, F (1, 390) = 560.07, 

p = .00. Thirty-eight percent of the variance was accounted for 

by procedural accuracy, 13% of the variance was accounted for 

by safety, and, trailing behind was efficiency, which accounted 

for only 6% of the variance. 
3

As seen in Figure 2, there is some evidence for flattening 

in the Safety measure, Efficiency is fairly level throughout the 

trials, and Procedural Accuracy declines steeply in Trial 3 but 

rebounds in Trial 4. 

Discussion

We examined the ATST performance data across trials to 

determine if any trials could be dropped or modified to shorten 

the ATST testing time. Results from this study suggested that 

participants continued learning how to perform the task during 

Trial 1 and did not reach performance stability by the end of Trial 

4. In addition, the level of difficulty across trials did not change 

in an incremental fashion, making it difficult to make meaningful between-trials comparisons. For example, the third trial has 

about twice as many planes per minute as the first two trials.

The results suggest that, in its current form, the ATST cannot 

be shortened by simply dropping testing trials without jeopardizing its reliability and validity. Therefore, we developed new 

instructions that incorporate guided practice sessions instead 

of trial-and-error-learning, as is currently employed, with the 

intention of stabilizing performance. If more stable performance 

occurs earlier, then perhaps one or more of the trials could be 

dropped, thereby reducing the time required to take the test. 

STUDY 2: DEVELOPMENT AND EVALUATION OF 

MODIFIED TESTING INSTRUCTIONS

Research in other disciplines such as computerized assessment 

has shown that effective test instructions can reduce training 

time without degrading performance. Rohlman, Sizemore, 

Anger and Kovera (1996) found that instructions that covered 

one step per screen produced better understanding of the task, 

as measured by improved performance, when compared with 

traditional instruction formats.

Based on the Rohlman et al. (1996) findings that the most 

effective instructions are interactive and provide one action per 

page, we developed modified instructions for ATST. 

Modified Instructions. We developed modified instructions 

for the ATST using a PowerPoint version of the ATST control 

screen to create an interactive learning experience for participants. The modified instructions consisted of PowerPoint slides 

adapted from various scenarios included in the ATST instructions. The slides were controlled by clicking interactive buttons 

that allowed the participant to proceed to the next section or 

return to the previous section. 

Sections of the modified instructions included: 1) what the 

ATST measures, 2) specific information to be familiar with, 3) 

a description of the airspace and control panel, which includes 

heading, speed, level, pilot read back window, landing headings, 

the five-mile scale marker, total time, and elapsed time, 4) objectives, 5) rules with accompanying demonstrations for each of 

the three types of errors– hand-offs (efficiency), separation, and 

procedures, 6) brief information about the number of sessions 

for the practice tests and the actual test, 7) brief information on 

how to use the headset, and 8) a slide presented at the end of 

the instructions that included two interactive buttons that gave 

the option of either repeating the instructions or proceeding to 

the practice scenarios. 

The individually-paced modified instructions described the 

task of each section, presented examples, and provided practice 

Figure 2. Performance percent correct across trials for Efficiency, Safety, and Procedures.

0

10

20

30

40

50

60

70

80

90

Trial 1 Trial 2 Trial 3 Trial 4

%Correct

Trials

Efficiency

Safety

Procedures
4

time. Based on an individual’s confidence with his/her competency on a task, a participant could either repeat a section or 

proceed to the next one. Samples of each type of instructions 

are shown in Appendices B and C.

The original ATST instructions were presented with multiple 

actions per page and did not provide an opportunity to practice 

the actions. For this study, the standard ATST instructions were 

converted to a PowerPoint format that replicated the paging and 

layout of the original instructions.

For both modified and standard instructions, we created a 

timer program that logged the amount of time spent reading 

the instructions and launched the ATST program at the end of 

the instructions. 

In Study 2, we collaborated with the USAF to test the effectiveness of the modified instructions. To determine whether modified instructions would reduce testing time through improved 

performance, we compared the performance of two groups of 

participants receiving either modified or standard instructions. 

We examined how much time each group spent reading the 

instructions to determine whether the new instructions would 

be equivalent to the old instructions. In addition, we compared 

ATST performance scores for the two groups. We hypothesized 

that participants receiving modified instructions would perform 

better than those receiving standard instructions. We thought 

that the modified instructions, by providing some practice with 

the control panel and mouse, would improve scores, especially 

in the first trial.

We also hypothesized that the performance scores of those 

receiving modified instructions would have reduced variability, 

earlier in the test, as compared with the performance scores of 

those receiving standard instructions. Again, we felt that practice 

would produce better scores and therefore lower variability.

Method

Participants. We entered into an agreement with the USAF 

to compare the efficacy of the ATST modified instructions 

with that of the standard instructions. Data were collected 

during the period June 25 through September 10, 2011, from 

242 USAF recruits enrolled in basic training at Lackland AFB, 

San Antonio, TX. The average age of the recruits was about 22 

years. The sample was 74% male and 26% female. Participants 

were a diverse sample: ethnically – 71% were non-Hispanic or 

non-Latino, 15% were Hispanic or Latino, and 14% declined 

to respond; racially – 76% were white, 15% were black/African 

American, 2% were Asian, 4% were Other, and 3% declined to 

respond. Most (96%) recruits in our sample had earned some 

college semester hours. 

Design and Data Analysis

The design of the study was a mixed-model, randomized 

design in which instruction type (modified versus standard) 

represented the between-groups variable, trials represented the 

within-subjects variable, and performance measures represented 

the repeated measures. Because we were measuring several different dependent variables (DVs) at several different times, we 

chose repeated measures multivariate analysis of variance analysis 

to prevent inflation of the probability of making a type I error, 

that is, finding significant results by chance (Tabachnick & 

Fidell, 2007). 

Procedure

The instructions and timer programs were beta tested before 

mailing the packaged suite of instructions, timer programs, ATST 

program, and installation guidelines to the USAF’s contractors 

in San Antonio, TX. The testing package was installed on 20 

computers at the Lackland Air Force Base Applied Performance 

Assessment and Testing (APAT) facility. Half of the computers 

were randomly selected to have the modified instruction package installed, while the other half had the standard instruction 

package installed. Air Force recruits, naïve about ATST, were 

briefed about the study by USAF contract proctors. Data were 

collected each week on Saturdays for 12 weeks, from June 26, 

2011, to September 10, 2011.

Data were screened and means and standard errors were 

computed for three ATST criterion measures (efficiency, safety, 

and procedural accuracy). Following the data screening, we 

conducted a multivariate repeated measures analysis of variance. 

Additional analyses were conducted to compare whether there 

were differences in the performance of Air Force recruits versus 

FAA applicants (from Study 1). Because the modified instructions were not developed until Study 2, only data from recruits 

who received the standard instructions were compared with data 

from FAA applicants.

Results

The original Air Force candidate sample consisted of 242 recruits. Changes were made to the testing protocol after the start 

of the study, and the first 20 cases were dropped. An additional 

29 cases were dropped as outliers or containing missing data, 

resulting in a final sample size of 193. The demographics of the 

193 cases were not different from the full sample.
5

Air Force Recruits

The modified instructions took significantly longer to read 

than the standard instructions, t = 13.16, p = .000. Figure 3 

presents the distribution of time spent reading the instructions. The mean length of time (measured in seconds) it took 

recruits to finish reading the modified instructions was 770.16 

sec. (approx. 13 min) with a standard deviation of 247.12 sec. 

(approx. 4 min) for 104 recruits. The mean time to finish reading the standard instructions was 427.75 sec. (approx. 7 min) 

and a standard deviation of 83.04 sec. (approx. 1.5 min) for 100 

recruits. The slowest reading times were approx. 23 minutes for 

the modified instructions, versus approximately 11 min for the 

standard instructions. 

Figure 3. Length of time (duration in seconds) for USAF recruits to read the instructions.

0

5

10

15

20

25

30

35

40

45

50

Frequency

Duration (seconds)

MOD

STD
6

Our first hypothesis was that participants receiving modified 

instructions would perform better than those receiving standard 

instructions. As seen in Figure 4, there was no difference in 

overall performance between those receiving the standard and 

modified instructions (i.e., there was no main effect of instruction 

type; efficiency, F (1, 202) = 0.01, p = .940; safety, F (1, 202) = 

1.10, p = .296, and procedural accuracy, F (1, 202) = .144, p = 

.705). For the efficiency measure, the means and standard errors 

for the modified instructions and standard instructions were, 

respectively, MM= 55.30, SE= 1.71 and MS= 55.20, SE= 1.68. 

Means and standard errors for the safety measure, respectively, 

for both instruction types were MM=58.86, SE=1.67 versus 

MS=61.06, SE=1.65. Procedural accuracy means and standard 

errors were MM=39.99, SE=1.79 for the Modified Instructions 

and MS= 41.25, SE=1.76 for the Standard Instructions. Thus, 

this hypothesis was rejected. 

Our second hypothesis was that the performance scores of 

those receiving modified instructions would have reduced variability, earlier in the test, as compared with the performance 

scores of those receiving standard instructions. Figure 4 shows 

that there was no trials by instruction type interaction, (F (9, 

194) = .824, p = .60); therefore, Hypothesis 2 was not supported 

Figure 4. Average performance for each ATST measure by instruction type. Error bars represent the 

standard errors.

0

10

20

30

40

50

60

70

80

Efficiency Safety Procedural

Accuracy

Average Performance Scores

MOD

STD

either. Moreover, results indicate that there were no significant 

effects between the two instruction types at the univariate level – 

efficiency, F (1, 191) = .002, p = .968; partial eta squared = .000; 

safety, F (1, 191) = .877, p = .350, partial eta squared = .005; 

and procedural accuracy, F (1, 191) = .253, p = .616, partial eta 

= .001. Thus, there is no indication that performance was stable 

across trials for either instruction type.

Performance comparison of Air Force Recruits 

versus FAA applicants

Because the performance of the USAF recruits was lower 

than that of the applicants for FAA ATCS positions whose data 

were analyzed in Study 1, we felt that these results might not 

generalize to our population of interest, i.e., FAA and USAF air 

traffic control applicants. Therefore, we conducted an analysis 

comparing the performance of the USAF recruits with that of the 

FAA applicants. We interpreted these results with great caution, 

recognizing that the data were collected under different circumstances and for different purposes. We did this only to determine 

if the performance of the USAF recruits was sufficiently similar 

to that of FAA applicants to establish whether the data can be 

used to assess the effectiveness of the Modified Instructions. 
7

As indicated in Figure 5, the profile analysis resulted in a 

main effect of data source (FAA v USAF), F (1, 494) = 195.15, 

p =.0001, partial eta squared = .247, suggesting that the performance of the USAF participants was different from that of FAA 

applicants. Additionally, the source by scores by trials interaction 

was significant, F (4.07, 2421.52)1 = 6.09, p= .0001, partial eta 

squared = .011. From this, we determined that the AF sample 

pattern of performance was different from the FAA sample. 

This is especially noticeable for the Procedures and Efficiency 

measures. We conclude that the USAF sample was not similar 

to the sample of FAA applicants. 

Discussion

The present study was conducted because the USAF requested 

a shorter version of the ATST that does not jeopardize the reliability and validity of the test. For the test to be operationally 

more useful for the Air Force, the test would need to take less 

time to complete. We first conducted a preliminary analysis of 

data collected from applicants for FAA ATCS positions to determine if any testing trials could be dropped to shorten the ATST 

testing time. We did this by examining the ATST performance 

data across trials. Results suggested that participants continued 

learning to perform the task during Trial 1 and did not reach 

performance stability by the end of Trial 4. Because it appeared 

that learning continued throughout the trials, it did not seem 

feasible to shorten the ATST by eliminating trials. Thus, we 

considered changing the way instructions for taking the test 

were given in an attempt to reduce the time participants take 

to reach stable performance before beginning the testing phase. 

We thought that this could be accomplished through improved 

instructions. We constructed a set of modified instructions with 

the goal of attaining more stable performance earlier in the 

ATST testing session.

Subsequently, we proposed two hypotheses: 1) Participants 

receiving modified instructions would perform better than those 

receiving standard instructions; 2) Performance scores of those 

receiving modified instructions would have reduced variability, 

earlier in the test, as compared with the performance scores of 

those receiving standard instructions. 

Hypothesis 1

We expected that participants receiving modified instructions 

would perform better than those receiving standard instructions. We saw no differences in performance (either overall or 

for specific ATST measures) between groups receiving different 

types of instructions within or across trials. The improvement in 

performance that we had hoped for was not found, but because 

performance did not decline indicates that the approach used 

to develop the modified instructions has utility.

Hypothesis 2

We had hypothesized that our modified instructions would 

result in more stable performance across trials. We did not 

observe a trials by instruction type interaction on either overall 

Figure 5. ATST estimated performance mean scores for measures of efficiency, safety, and 

procedural accuracy by trial and group — USAF recruits versus FAA applicants.

0

10

20

30

40

50

60

70

80

90

Trial 1 Trial 2 Trial 3 Trial 4

Average Performance Scores

Trials

Efficiency FAA

Efficiency AF

Safety FAA

Safety AF

Procedures FAA

Procedures AF
8

performance scores or individual performance scores for those 

receiving modified instructions, as compared to those receiving 

standard instructions. In other words, the groups did not differ 

in performance across trials. 

Performance Comparison

We visually observed that the performance of the participants 

in Study 2 appeared to be worse than that of FAA applicants on 

all measures across all trials. Therefore, we conducted an analysis 

comparing the performance of the USAF recruits with that of 

the FAA applicants. Interpreting these results with great caution, 

we found that the performance of the USAF participants was 

significantly lower than that of the FAA applicants. Furthermore, 

the three-way interaction suggests that there were differences in 

performance beyond purely poorer performance. We concluded 

that the USAF sample is not similar to the sample of FAA applicants. This lack of similarity suggests that task engagement 

(or lack thereof) may be a reasonable explanation for the lack of 

difference between the two instruction types. FAA applicants are 

likely to be more motivated than the Air Force recruits because 

they were taking the test for the benefit of getting a job; Air Force 

participants were volunteers who already had a job. 

Subsequent efforts to reduce the time allowed for ATST administration would require testing of participants more similar 

to our applicants. Additional research with a more motivated 

sample of participants would seem warranted. Based on our 

results, we now have some information that can guide future 

research. We should develop different scenarios that are equal 

in length and equal in difficulty to determine the effect of the 

instructions, if any. These scenarios should be shorter to detect 

the point at which changes in performance due to learning are 

no longer apparent. From our analysis, we know that the Modified Instructions take more time than the Standard Instructions, 

but this may be an artifact of the animations and opportunities 

for practice in the modified instructions. Future research will 

be necessary to determine if a more interactive, practice-based 

approach to instructions is preferable to the ATST instructions 

used now.

REFERENCES

Broach, D., & Brecht-Clark, J. (1993). Validation of the 

Federal Aviation Administration air traffic control 

specialist Pre-Training Screen. Air Traffic Control 

Quarterly, 1, 115-133.

Carretta, T.R., & King, R.E. (2008). Improved military air 

traffic controller selection methods as measured by 

subsequent training performance, Aviation, Space, and 

Environmental Medicine, 79, 36-43.

Nickels, B.J., Bobko, P., Blair, M.D., Sands, W.A., & Tartak, 

E.L. (1995). Separation and control assessment (SACHA) 

final job analysis report. Bethesda, MD: University 

Research Corporation. 

Ramos, R.A., Heil, M.C., & Manning, C.A. (2001). 

Documentation of validity for the AT-SAT computerized 

test battery, Volume I. (Report No. DOT/FAA/

AM-01/5). Washington, DC: Federal Aviation 

Administration Office of Aviation Medicine.

Rohlman, D.S., Sizemore, O.J., Anger, W.K., & Kovera, C.A. 

(1996). Computerized neurobehavioral testing: 

Techniques for improving test instructions. 

Neurotoxicology and Teratology, 18(4), 407-412.

Scarborough, A.L., & Bleckley, M.K. (May 2008). Assessing 

learning across trials in the air traffic scenarios test 

(ATST). Presented at the 87th Annual Scientific Meeting 

of the Aerospace Medical Association, Boston, MA.

Tabachnick, B.G., & Fidell, L.S. (2007). Profile analysis: The 

multivariate approach to repeated measures (Chapter 8). 

In Using multivariate statistics (5th ed.). New York: 

Harper Collins, pp. 311-374.

Weltin, M., Broach, D., Goldbach, K., & O’Donnell, R. 

(April, 1992). Concurrent criterion-related validation of air 

traffic control specialist Pre-Training Screen. (Final report 

submitted under FAA contract DTFA01-89-Y-01019). 

Washington, DC: FAA Office of the Deputy 

Administrator, ADA-001.
A1

APPENDIX A 

ATST Practice and Test Duration (in minutes)

APPENDIX A – ATST Practice and Test Duration (in minutes)

B1

APPENDIX B 

An Example of the ATST Standard Instructions

APPENDIX B – An Example of the ATST Standard Instructions

CONTROLS

You give directions to the planes on your screen by first 

moving the mouse cursor (white arrow) over the plane you 

want to control and then pressing (clicking) the left mouse 

button.

You must click on the plane itself (the green arrow) and NOT on 

the plane’s data block. Clicking on the data block will have no 

effect.

When a plane is ready to receive an instruction, the plane and 

its data block will turn yellow. Only one plane can receive an 

instruction at a time. Clicking on a second plane while another 

plane is highlighted in yellow will have no effect.

After you click on a plane and it turns yellow, you MUST select 

ONE of the instructions in the control section to the right of the 

screen.

Click on “Next” to continue,

Or “Previous” to go back.

C1

APPENDIX C

An Example of the ATST Modified Instructions

22

APPENDIX C – An Example of the ATST Modified Instructions

A control panel is

used to change the

heading, speed, and

level.
C2

 The Heading menu shown by the white

arrow is used to change aircraft directions. 

There are eight headings 0-7:

• 0 and 4 control directions 

 up (0) and down (4)

• 2 and 6 control directions 

right (2) and left (6)

• 1, 3, 5, and 7 are for diagonal headings

 

• The aircraft will fly in the direction

you select until it exits, crashes, or 

 you change the direction again.
C3

 Example:

This aircraft (green arrow) 

has a Heading 4. To change 

direction, use the mouse pointer

to click on the aircraft. When the 

arrow turns yellow, select the 

Heading 2 for the aircraft.

 M2f (Click on the arrow)
C4

Now practice changing the direction of

the aircraft using the Heading menu of

the control panel. When you are 

satisfied that you know how to change 

headings, click on the Next button to go 

to the Speed menu. To review the example, 

click on the Previous button.

(Click on the arrow)