DOT/FAA/AM-00/30

Scott H. Mills

Civil Aeromedical Institute

Federal Aviation Administration

Oklahoma City, Oklahoma 73125

September 2000

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

The Computerized Analysis of

ATC Tracking Data for an

Operational Evaluation of

CDTI/ADS-B Technology

Office of Aviation Medicine

Washington, D.C. 20591

U.S. Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-00/30

4. Title and Subtitle 5. Report Date

The Computerized Analysis of ATC Tracking Data for an Operational September 2000

Evaluation of CDTI/ADS-B Technology 6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Mills, S.H.

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

FAA Civil Aeromedical Institute

P.O. Box 25082

Oklahoma City, OK 73125 11. Contract or Grant No.

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aviation Medicine

Federal Aviation Administration

800 Independence Ave, S.W.

Washington, DC 20591 14. Sponsoring Agency Code

15. Supplemental Notes

This work was performed under task AM-B-00-HRR-516

16. Abstract

In 1999, the Cargo Airlines Association and the Federal Aviation Administration conducted an operational

evaluation (OpEval) of Cockpit Display of Traffic Information (CDTI) and Automatic Dependent

Surveillance - Broadcast (ADS-B) technologies at the Airborne Express Airpark in Wilmington, Ohio. This

evaluation was designed to demonstrate the benefits of CDTI, including safety, efficiency, and capacity. The

evaluation included 13 aircraft of various types and their flight crews. The aircraft flew multiple flight

patterns during the morning and the afternoon of a single day. Each traffic pattern flown by each aircraft was

assigned to either the CDTI or baseline (no CDTI) condition. Human factors observers recorded data from

the flight decks and the control tower. In addition, air traffic control (ATC) data were recorded by the

participating ATC facilities. An important part of the analysis of such a demonstration is the examination of

objective flight data. Because of the complexity of the OpEval, new computerized analysis techniques were

developed and conducted. This paper describes those techniques in detail, as well as the results of the analysis.

Methods such as those described here will be increasingly important as new technologies are developed and

evaluated operationally.

17. Key Words 18. Distribution Statement

CDTI, ADS-B, Operational Evaluation, Radar Document is available to the public through the

National Technical Information Service,

Springfield, Virginia 22161

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 12

1

THE COMPUTERIZED ANALYSIS OF ATC TRACKING DATA FOR AN OPERATIONAL

EVALUATION OF CDTI/ADS-B TECHNOLOGY

INTRODUCTION

The availability of new technologies for the cockpit and air traffic control facilities is creating new

capabilities for enhanced aircraft operations and,

with them, the need to evaluate the effectiveness of

these new technologies in operational settings. Two

such systems, Cockpit Display of Traffic Information (CDTI) and Automatic Dependent Surveillance

- Broadcast (ADS-B), were recently demonstrated in

an operational evaluation (OpEval) conducted at the

Airborne Express Airpark in Wilmington, Ohio. The

OpEval was sponsored by the Cargo Airlines Association (CAA) and the SafeFlight21 Office of the Federal Aviation Administration (FAA), and included

aircraft and flight crews from industry, government,

and private organizations.

The purpose of the OpEval was to demonstrate the

use and expected benefits of CDTI, which consist of

increased safety, efficiency, and capacity. Some specific examples of these benefits are: enhanced visual

acquisition for “see and avoid,” enhanced visual

approaches, and efficient departure and final approach spacing. An important part of this evaluation,

from an ATC human factors perspective, was the

analysis of objective flight data to quantify and confirm the demonstrated benefits of the new technologies. However, because of the complex nature of the

OpEval, innovative field assessment techniques had

to be developed and conducted to accomplish that

purpose. This paper describes the development of these

assessment procedures and the resulting analysis.

METHOD

The OpEval was conducted on July 10, 1999, at

Airborne Express Airpark in Wilmington, Ohio.

Thirteen aircraft of different types (primarily B727’s

and DC-9’s) were involved in flying multiple traffic

patterns using two parallel runways, 22L and 22R

(see Figure 1). Flights took place between 0900-1100

hours and between 1300-1600 hours (local time).

Each pattern for each aircraft was assigned to either

the CDTI or baseline (no CDTI) condition, according to the experimental design. A total of 168 patterns were flown throughout the day. All aircraft were

in radio contact with Dayton TRACON and

Wilmington Tower. Air traffic controllers from both

facilities provided instructions and clearances as

needed.

Figure 1. Flight Profiles
2

Both subjective and objective data were recorded

as part of the OpEval. Human factors observers were

stationed on the flight decks of participating aircraft

and collected various types of data, including crew

response times to air traffic control (ATC) communications and crew interaction with CDTI systems.

Observers were also present in the Dayton TRACON

to assess the effects of CDTI use on subjective controller workload. ATC system data were routinely

recorded by the control facilities involved: Radar and

voice communication data were recorded by Dayton

TRACON, and tower voice communications were recorded by Wilmington Tower. Indianapolis Air Route

Traffic Control Center recorded radar data for activities

occurring outside the TRACON’s airspace. These data

were made available to researchers for analysis.

Data Analysis

An important objective of this analysis of ATC

data was to develop databases of aircraft locations (as

indicated by radar), as well as important characteristics of each traffic pattern flown, such as when events

occurred (e.g., crossing the runway threshold on

completion of a pattern). These databases were used

as input for other analyses, such as the measurement

of aircraft pair distances at the time of approach

clearances.

Another objective of this analysis was to obtain

measures of accuracy and variance for CDTI-based

spacing during the stage of flight known as the final

approach. Comparison of these measures with those

collected during the baseline condition can provide

evidence as to the effectiveness of CDTI use. Accomplishing this objective required the identification of discrete spacing events, including specific

times and aircraft pairs. Additionally, the distances

between aircraft during the events had to be calculated. Achieving both of the analysis objectives, database construction and final approach spacing, required

the development of a series of procedures, described

in detail below.

Three primary data sources provided the basis for

analyses. These included ATC radar system recordings provided by Dayton TRACON, transcripts of

all voice communications between the TRACON

and participating aircraft (in the form of electronic

spreadsheets), and records that indicated under which

condition—CDTI or baseline—each aircraft flew on

each pattern. Information from these sources was

used to accomplish the objectives of the analysis, as

described in the steps below.

Step 1: ATC Reduction of Radar Data.

The radar system recordings provided by Dayton

TRACON had to be further processed by an FAA

National Airspace System software program, “CDR

Editor,” on an Automated Radar Terminal (ARTS)

mainframe computer. Because the volume of data for

this analysis was so great, it was not practical to

process the data at the Dayton facility. Such processing requires the allocation of significant memory and

processing resources on the ARTS computer; because

TRACON facility computers continuously process

live traffic data, it was not possible to use one of them

for such a task. Consequently, the OpEval data were

processed by the ARTS computer located at the FAA

Academy in Oklahoma City. This system is routinely

used for training and instructional purposes.

The CDR Editor produced approximately 4.3

million lines of textual output, consisting mainly of

aircraft position reports and ATC system messages.

The output was transferred to a Windows NT-based

computer network, upon which subsequent analyses

were performed (See Figure 2). The Windows-based

computer platform allowed maximum flexibility for

researchers to use application software such as spreadsheet and database programs and to develop the

customized software (written in Visual Basic) that

performed much of the subsequent analysis. The

CDR Editor output was also transferred to a Unixbased computer network for use with the Systematic Air

Traffic Operations Research Initiative (SATORI) software, designed to utilize TRACON data. (Rogers, 1993).

Step 2: Development of Aircraft Location and

Pattern Events Databases.

To perform analyses involving aircraft positions,

databases were needed that contained information

about the recorded locations of all participating aircraft throughout the OpEval. In addition, a database

was needed that defined each pattern flown by each

aircraft and the important events within each pattern.

Once developed, these databases served as the basis

for further analyses, as well as for archival purposes.
3

Radar Track Records for Relevant Aircraft. To develop the database of aircraft locations, all radar

tracking records for the participating aircraft had to

be identified. These records were produced each time

the ATC radar sweep made contact with an aircraft.

The CDR Editor output data consisted of all

TRACON system messages and radar tracking reports recorded from two radar sites (Dayton and

Wilmington) during an eight-hour period that encompassed the OpEval. Radar tracking reports comprise several data fields, including time of day, aircraft

identification code (ACID), assigned beacon code

(ABC), reported beacon code (RBC; reported by the

aircraft’s transponder), and x,y coordinates, relative

to the radar site (see Figure 2).

Software was developed that extracted tracking

records for which the ACID matched one of the

participating aircraft in the OpEval. However, inspection of the results of this procedure revealed that

no records existed for participating aircraft at altitudes below 1500 feet (mean sea level), even though

the field elevation at the airpark was approximately

1100 feet. Further examination showed that tracking

records did exist for aircraft at altitudes below 1500

feet, although the ACID data fields were blank.

These records did, however, have values in the ABC

field, which could also be used to identify aircraft.

To identify an aircraft by the ABC, it is necessary

to know which beacon code had been assigned to that

aircraft at the time of the data record. Knowing the

specific time is necessary because the beacon code

assignment for a particular aircraft can be changed

during the day (see Figure 3 for examples of these

characteristics of the CDR editor output). Therefore,

software was written that scanned the CDR Editor

output and determined which beacon codes were

assigned to which aircraft at specific times. This was

done by sorting the records that had values in the

ACID field by time and then determining the periods

during which each ACID was assigned to specific

beacon codes. Once this was accomplished, other

software extracted all records that corresponded to

participating aircraft by matching either on ACID or

on ABC. This process resulted in a complete database, coded by ACID, of tracking records for all

aircraft involved in the OpEval.

TRACKING DATA 7/10/99 PAGE 71500

TIME ACID ABC RBC FRM RALT X Y HDG SPD ...

19:56:16.445 255 1200 1200 38 3500 -13.69 -19.19 266 90 ...

19:56:16.445 112 1200 1200 19 2400 11.25 46.81 256 57 ...

19:56:16.446 FDX9002 143 4515 4515 30 2600 -1.06 -1.25 191 188 ...

19:56:16.447 UAL1288 18 1364 1364 38 8600 5.38 17.38 175 294 ...

Figure 2. CDR Editor Output - Tracking Records

TRACKING DATA 7/10/99

TIME ACID ABC RALT

10:45:43.978 FDX9002 .. 4276 1800

10:45:48.474 FDX9002 .. 4276 1700

14:07:39.335 FDX9002 .. 4204 1500

14:07:43.731 FDX9002 .. 4204 1500

14:07:48.123 .. 4204 1400

14:07:52.618 .. 4204 1400

14:08:23.380 .. 4204 1300

14:08:27.680 FDX9002 .. 4204 1500

18:11:42.500 UPS606 .. 4276 2900

18:11:47.087 UPS606 .. 4276 2900

Aircraft has

different assigned

beacon codes at

different times.

Aircraft ID is not

present when

altitude < 1500.

Same beacon code is assigned

to different aircraft at

different times.

Figure 3.. CDR Editor Output – Aircraft Identification
4

Development of the Pattern Events Database. Many

of the analyses planned for the OpEval required the

development of a database that described key events

and characteristics of each pattern flown. Examples

of these include the times the aircraft began and

completed the pattern, the pattern altitude, and the

experimental condition (CDTI or baseline). The

determination of these values was accomplished in a

variety of ways as described below. As events and

characteristics were identified, the related information was entered into a database either electronically

or manually, depending on the method of identification (See Figure 4).

Several events (Departure, Downwind Leg, and

Start of Descent) were identified by scanning the

altitude profile of each aircraft. To do this, the

database of tracking records was sorted by ACID and

then by time of day. The beginning of the Departure

stage for each pattern was obtained by scanning these

records. The end of the Departure stage and the

beginning of the Downwind Leg were similarly determined and defined as the point at which the

aircraft reached pattern altitude (for that pattern,

altitudes were assigned by ATC and were sometimes

changed from one pattern to the next). Start of

Descent was identified as the point at which the

aircraft began descending from pattern altitude.

Approach clearance times were manually extracted

from the voice communications transcripts. This was

accomplished by searching for keywords such as

“cleared,” as in the ATC message, “...CLEARED ILS

TWO TWO RIGHT APPROACH.” Another key

event was the time at which the aircraft passed the

outer marker. The outer marker is a navigational aid

that aircraft typically cross after they have been cleared

for runway approach. Outer marker crossing times

were determined by software that tracked the aircraft’s

locations (using x,y coordinates) from the time of the

approach clearance to the time it crossed the coordinate location of the outer marker for the appropriate

runway. Similarly, runway threshold crossing time

was determined by tracking the aircraft from the time

of crossing the outer marker to the coordinate location of the threshold of the correct runway. Figure 5

illustrates several of the these events for a hypothetical flight pattern.

Step 3: Identification of Approach Events.

To measure the spacing performance of aircraft

during the final approach stage of flights, it was

necessary to identify when specific aircraft pairs were

both on final approach and lined up with the same

runway. These periods will be called approach events.

An approach event was identified for a trailing and

Pattern ACID Runway Condtn Dep Level Ptrn Alt Clear Marker Threshold ...

1 FDX9001 22L baseline 13:27:17 13:29:19 4000 13:36:23 13:39:30 13:41:07 ...

2 FDX9001 22L CDTI 13:41:07 13:43:13 4000 13:48:34 13:51:59 13:53:45 ...

...

1 ABX33 22R CDTI 18:03:58 18:05:34 3000 18:08:57 18:08:59 18:09:04 ...

2 ABX33 22R CDTI 18:12:07 18:13:11 3000 18:18:09 18:20:23 18:22:48 ...

Figure 4. Pattern Events Database

Figure 5. Stages of Flight

Downwind

Departure

Approach

Clearance

Outer Marker

Runway Threshold
5

leading aircraft pair from the time both were aligned

with the same runway until the lead aircraft crossed

the runway threshold. (see Figure 6.)

The identification of the OpEval approach events

was accomplished with the use of the TRACON

SATORI system. SATORI presents a graphical recreation of TRACON radar information, including

the synchronized replay of voice communications.

This system allowed researchers to review aircraft

movements and identify approach events. The identity of the aircraft pairs plus the start and end times

of each event were then entered into a database for

further analyses.

Step 4: Calculation of Aircraft Locations and

Distances.

To measure the accuracy of aircraft spacing, the

distance between each trailing and lead aircraft during each approach event had to be calculated.

Therefore, the coordinate locations of each aircraft at specific times during the event had to be

obtained. The radar antenna at the Dayton TRACON

sweeps the airspace in a circular direction approximately every five seconds, resulting in a position

report for each aircraft each five seconds. Figure 7

illustrates how this tracking procedure results in

position reports at different times for different aircraft during an individual sweep. Therefore, direct

distance calculations from the tracking records are

not possible. To address this situation, software was

developed that uses linear interpolation to obtain the

coordinate positions of aircraft at specific times.

These positions were then used to calculate distances

between aircraft during each approach event.

Step 5: Production of Graphical Profiles.

To enable a better understanding of OpEval aircraft dynamics, the TRACON Radar Charting System (TRACS) was developed. TRACS is a software

system that allows researchers to view graphical radar

profiles of flights that occurred during a specified

time interval, such as the OpEval. The program

presents the movements of data blocks on a screen.

The textual information in the blocks indicate aircraft identity and altitude, and the position of the

blocks indicates lateral position. It is also capable of

displaying symbols that represent the histories of the

movement of the aircraft.

TRACS can also use information from the Pattern

Events database to graphically code these histories to

indicate the occurrence of key events, such as approach clearances, or to indicate characteristics of

individual patterns, such as the experimental condition in effect. In addition, the screen output from

TRACS can be printed to provide permanent records

of these profiles.

Figure 6. Approach Event Identification.

Approach Event

Approach Event ends when lead aircraft

crosses runway threshold.

Approach Event has not started.

Aircraft 1: Position

recorded at 00:00:00

Aircraft 2: Position

recorded at 00:00:01

Figure 7. Radar Aircraft Position Recordings
6

RESULTS

Many of the results derived from the analyses of

tracking data are reported and discussed in the OpEval

Final Report (Operational Evaluation Coordination

Group, in press).

Several results that emerged specifically from the

analysis of the tracking data are presented here. A

comprehensive review and discussion of the OpEval

findings are available in the referenced report. Because of the demonstrational nature of the OpEval,

inferential statistical analyses were not conducted on

these data. However, analyses involving descriptive

statistics may provide some insight into the effects of

CDTI use during the OpEval, and are therefore

reported below.

Approach Event Occurrence

A total of 88 approach events occurred during the

OpEval. Of these, 47 occurred in the morning, and

41 occurred in the afternoon. The duration of the

events ranged from a minimum of 22 s to a maximum

of 160 s (See Table 1).

Approach Event Spacing

Because of low visibility conditions in the morning, those approaches were flown using instrument

landing systems (ILS), as opposed to the afternoon,

during which all approaches were visual. Because of

this significant difference in flying conditions, spacing measurements are presented separately for morning and afternoon approaches. Table 2 includes means

and standard deviations for approach event spacing.

Figures 8 and 9 depict the frequency distributions of

spacing distances for the baseline and CDTI patterns,

flown in the morning and afternoon, respectively.

For a number of reasons it is difficult to form

conclusions about differences in spacing performance

between the baseline and CDTI conditions. The

required use of ILS in the morning probably reduced

the opportunity for flight crews to use CDTI, especially during the last portion of final approach, when

spacing was determined by ATC instructions and the

Figure 8. Frequency distribution of spacing measurements for the AM session.

Spacing Distance - AM

0

20

40

60

80

100

456789 10 11 12 13

Nautical Miles

Frequency

Baseline

CDTI

Table 1. Approach Event Occurrence

and Durations.

Approach

Event

Number of

Approach

Events

Mean

Duration of

Events

(Seconds)

Morning

Baseline 18 82.9

CDTI 29 81.1

Afternoon

Baseline 6 72.5

CDTI 35 65.8

Table 2. Approach Event Spacing.

Approach

Event

Mean Standard

Deviation

Morning

Baseline 7.3 1.2

CDTI 6.8 1.8

Afternoon

Baseline 4.6 1.2

CDTI 3.6 1.4

(nautical miles)
7

Spacing Distance - PM

0

20

40

60

80

100

1.5

2

2.5

3

3.5

4

4.5

5

5.5

6

6.5

7

Nautical Miles

Frequency

Baseline

CDTI

Figure 9. Frequency distribution of spacing measurements for the PM session.

Figure 10. TRACS output—all morning flights.
8

use of the ILS. During the afternoon, comparisons

were also problematic because of the small number (6)

of baseline approach events, compared with CDTI

approach events (35). Nevertheless, the distributions

seem to indicate reduced spacing for the CDTI conditions during both morning and afternoon. The smaller

spacing distances observed for the afternoon patterns,

compared with those in the morning, are expected

because crews were able to perform visual approaches.

Graphical Flight Profiles

Figures 10 and 11 depict output samples from the

TRACS program. Figure 10 illustrates all morning

OpEval flights, coded by baseline or CDTI condition. Each small circle on the display represents a

tracked aircraft position. Each larger circle indicates

an aircraft’s position at the time it was issued an

approach clearance by ATC.

Figure 10 reveals that in general, approach clearances were issued for runway 22R (top runway in

figure) earlier in the pattern than those issued for

runway 22L. It also appears that more aircraft flew in

the CDTI condition on runway 22L, and more flew

in the baseline condition on runway 22R. Both of

these observations are examples of the types of information that could be valuable during the course of

data analysis.

Figure 11 represents all afternoon flights, also

coded by condition. Comparison of the afternoon

flight profiles with those from the morning illustrates

Figure 11. TRACS output – All afternoon flights.

22R

22L
9

an interesting characteristic of the OpEval—that in

general, the shapes of patterns flown in the morning

conformed more to a “typical” flight pattern, such as

that depicted in Figure 5, than did the patterns flown

in the afternoon. Afternoon pattern shapes were

markedly more variable than those from the morning, perhaps due to differences in ATC instructions

associated with visual approaches.

CONCLUSION

The introduction of new technologies in aviation,

such as improved communication, navigation, and

surveillance systems, is leading to an evolution of the

complex interaction between aircrews and ATC.

While these technologies offer significant benefits to

users of the aviation system, their applications must

be evaluated operationally prior to widespread

deployment. Programs such as the demonstration

conducted in Wilmington will become increasingly

important, as will the analyses of objective flight data

from such operations. The evaluation of the large

volume of data generated by such complex demonstrations is challenging, but it can be accomplished

by combining modern software tools with the development of new procedures and analysis techniques.

REFERENCES

Operational Evaluation Coordination Group (2000).

CAA/FAA ADS-B / Safeflight 21 Phase I – Operational Evaluation Final Report. (in press).

Rogers, M.D., & Duke, D.A. (1993). SATORI:

Situation Assessment Through Re-creation of

Incidents. The Journal of Air Traffic Control,

35(4), 10-14.
10