Kurt M. Joseph

Aviation Human Factors Consultant

Austin, TX 78731

David Domino

MITRE CAASD1

McLean, VA 22102

Vernol Battiste

NASA Ames Research Center

Moffett Field, CA 94035

Randall S. Bone1

B. Oscar Olmos1

February 2003

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

Office of Aerospace Medicine

Washington, D.C. 20591

A SUMMARY OF FLIGHTDECK

OBSERVER DATA FROM

SAFEFLIGHT 21 OPEVAL-2

DOT/FAA/AM-03/2
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
i

Technical Report Documentation Page 

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. 

DOT/FAA/AM-03/2 

4. Title and Subtitle 5. Report Date 

A Summary of Flightdeck Observer Data From February 2003 

SafeFlight 21 OpEval-2 6. Performing Organization Code 

7. Author(s) 8. Performing Organization Report No. 

Joseph KM1, Domino D2, Battiste V3, Bone R2, and Olmos BO2

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 

1

Aviation Human Factors Consultant, Austin, TX 78731 

2

Mitre CAASD, McLean, VA 22102 

3

NASA Ames Research Center, Moffett Field, CA 94035 

11. Contract or Grant No. 

This work was performed under 

contract DTFA-02-01-P-05316. 

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered 

Office of Aerospace Medicine 

Federal Aviation Administration 

800 Independence Avenue, SW 

Washington, DC 20591

14. Sponsoring Agency Code 

15. Supplemental Notes 

Work was accomplished under approved task AM-B-00-HRR-516. 

16. Abstract 

In 2000, the Cargo Airlines Association and the Federal Aviation Administration conducted Operational 

Evaluation 2 (OpEval-2) of Cockpit Display of Traffic Information (CDTI) and Automatic Dependent 

Surveillance - Broadcast (ADS-B) technologies at the Standiford International Airport in Louisville, KY. This 

evaluation was designed to demonstrate the benefits of CDTI/ADS-B, including safety, efficiency, and capacity. 

Toward this end, five flight periods were designed to test five different applications: Airport Surface Situation 

Awareness (ASSA), Departure Spacing, Initial and Final Approach Spacing, Final Approach Runway 

Occupancy Awareness (FAROA), and Visual Acquisition and Traffic Awareness. OpEval-2 included 16 aircraft 

of various types and their flight crews. Flightdeck observers accompanied these flight crews to record human 

factors data. This report summarizes the collection and analysis of flightdeck observer data. In addition, the 

report includes empirically based recommendations for addressing human factors issues related to certification 

of CDTI for the five applications that were tested in OpEval-2. 

17. Key Words 18. Distribution Statement 

OpEval-2, CDTI, ADS-B, Human Factors, Flightdeck 

Observer Data, Situation Awareness 

Document is available to the public through the 

National Technical Information Service, Springfield, 

Virginia 22161

19. Security Classif. (of this 

report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price 

Unclassified Unclassified 60 

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

iii

ACKNOWLEDGMENTS

The first author thanks Dr. Roni Prinzo, who provided invaluable assistance and many

insightful comments while serving as Contracting Officer’s Technical Representative for

contract DTFA-02-01-P-05316.

1

A SUMMARY OF FLIGHTDECK OBSERVER DATA

FROM SAFEFLIGHT 21 OPEVAL-2

flightdeck observer data forms and flight crew questionnaires, the training of flightdeck observers, and

the collection and analysis of flightdeck observer data

for OpEval-2. In addition, a summary of human

factors issues related to CDTI is presented for each

OpEval-2 application based on the results of flightdeck

observer data analysis. Before these summaries are

presented, a brief overview of OpEval-2 procedures

and applications is given.

OPEVAL-2 PROCEDURES

Several procedures were implemented to ensure the

safety of flight operations at Standiford International

Airport during OpEval-2, which consisted of six,

three-hour flight periods that occurred between October 26th and October 30th of 2000. No observer

data was collected during the sixth flight period,

which comprised a limited demonstration to key

industry participants. This flight period is not discussed further. The flight periods were timed to

ensure that OpEval-2 occurred during low traffic

volume at the airport. A separate team of eight controllers, including a coordinator for both the Air

Traffic Control Tower (ATCT) and Terminal Radar

Approach Control (TRACON), was trained to provide standard air traffic control services for all OpEval2 aircraft.

The ATCT limited access to runway 17R/35L to

only the 16 aircraft participating in OpEval-2. In

addition, the ATCT isolated the airspace near the

airport for OpEval-2 air traffic to minimize disruption to normal flight arrivals and departures that used

runway 17L/35R. The controllers established two

traffic patterns for OpEval-2. The first pattern was

labeled Outer Covey, and it was restricted to the jet

aircraft that were participating in OpEval-2. The

second pattern was labeled Inner Covey, and it consisted of a wide variety of aircraft and equipage,

including general aviation aircraft. Traffic from these

two patterns was merged by the controllers at a point

20 miles from the 17R/35L runway threshold.

Table 1 displays the participating aircraft by flight

period, application, and equipment type. Surface

map capability is also indicated. The flight periods

were developed such that the systematic variation of

INTRODUCTION

The availability of new technologies for the

flightdeck and air traffic control facilities is creating

new capabilities for enhanced aircraft operations and,

with them, the need to evaluate the effectiveness of

these new technologies in operational settings. Two

such systems, Cockpit Display of Traffic Information

(CDTI) and Automatic Dependent SurveillanceBroadcast (ADS-B), were recently demonstrated in

Operational Evaluation 2 (OpEval-2), which was

conducted at Standiford International Airport in

Louisville, KY. OpEval-2 was sponsored by the Cargo

Airlines Association (CAA) and the SafeFlight21 Office of the Federal Aviation Administration (FAA),

and included aircraft and flight crews from industry,

government, and private organizations.

The purpose of OpEval-2 was to demonstrate the

use of CDTI and ADS-B and their expected benefits,

which include increased safety, enhanced capacity,

and greater efficiency. Three objectives were planned

for OpEval-2. The first objective was to develop and

evaluate avionics and procedural modifications needed

to support operational approval for the following

applications: Initial and Final Approach Spacing,

Departure Spacing, Final Approach and Runway

Occupancy Awareness (FAROA), and Airport Surface Situation Awareness (ASSA). A fifth application,

Visual Acquisition and Traffic Awareness, was tested

in OpEval-1, but also was included in OpEval-2 to

evaluate a specific procedure (see OpEval-1 Final

Report, 1999, for OpEval-1 summary). The visual

acquisition application included the use of call sign in

the traffic call-out to take advantage of the information available on the traffic display. The second objective was to use these applications to evaluate ADS-B

technology in a terminal area environment with air

traffic controllers. The third objective was to highlight the safety and efficiency benefits of ADS-B by

providing a limited demonstration to key industry

participants, including labor, airline operations, general aviation, and the FAA.

An important part of OpEval-2 was the collection

and analysis of flightdeck observer data to aid in

quantifying and verifying the expected benefits of

CDTI. This report summarizes the development of
2

Table 1. Opeval-2 aircraft classified by Flight Event, Application, and CDTI Type. 

 

Aircraft ID Flight 

Period 1

Flight 

Period 2 

Flight 

Period 3 

Flight 

Period 4

Flight 

Period 5 ASSA

FAROA

Air and 

Ground 

Departure 

Spacing 

Initial and 

Final Spacing 

Visual Acquisition/ 

Traffic Awareness

UPS 1 (Build 4) 

Boeing 727 

UPS101 

✓ ✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

UPS 2 (Build 4) 

Boeing 727 

UPS202 

✓ ✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

UPS 3 (Build 4) 

Boeing 727 

UPS303 

✓ ✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

UPS 4 (Build 4) 

Boeing 727 

UPS404

✓ ✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

Build 2/4 

FedEx1 (Build 2) 

Boeing 727 

FDX1 

 ✓ ✓ No 

Map ✓ ✓ ✓ ✓

FAA1 

Boeing 727 

N40 

✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

FAA2 

Convair 580 

N49 

✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

Basic 

Prototype 

FAA3 

Convair 580 

N39 

✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

Collins1

Sabreliner 

N50CR 

✓ ✓ ✓ ✓ No 

Map ✓ ✓ ✓ ✓

L3 

Cessna Citation V 

N189H 

✓ ✓ ✓ ✓ Moving 

Map ✓ ✓ ✓ ✓

Advanced 

Prototype 

Honeywell2

Beech King Air 

A90 

N52EL 

✓ ✓ ✓ Moving 

Map ✓ ✓

AOPA 

Beech Bonanza 

N7236W 

✓ ✓ ✓

North 

Up 

Map 

✓ ✓

DCA 

Cessna 210 

N624MT 

✓ ✓ ✓

North 

Up 

Map 

✓ ✓

CNS Aviation 

Piper Lance 

N31920 

✓ ✓ ✓

North 

Up 

Map 

✓ ✓

Volpe 

Piper Aztec 

327DR 

✓ ✓ ✓

North 

Up 

Map 

✓ ✓

MX20/GX60 

UPS AT3

Beech King Air 

C90 N89TM 

✓ ✓ ✓

North 

Up 

Map 

✓ ✓

1 The Collins Sabreliner had the advanced spacing algorithm, but not the Surface Moving Map. 

2

The Honeywell King Air had the Surface Moving Map, but not the advanced spacing algorithm. 

3

The UPS AT King Air was equipped with both CDTI Build2/4 and MX20 platforms. CDTI Build2/4 platform was used during Flight Event 

1. Both platform types were used during Flight Event 2.
3

procedures within and across these periods produced

an extensive set of data that was used to evaluate

OpEval-2 applications. The following paragraphs

briefly summarize each of the five flight periods in

which flightdeck observers collected data.

Flight Period 1

Operations during Flight Period 1 were conducted

on RWY 17R. The first 90 minutes of flight time

included three full-stops, one after each of the long

approach spacing profiles (i.e., closing to 5 miles or

120 seconds at the runway threshold). Table 1 displays the participating aircraft by flight period, application, and equipment type. Surface map capability is

also indicated. Table 1 shows that the three FAA

aircraft participated, as did the L3 and Collins business jets. The remaining 90 minutes of flight time

consisted of low approaches and use of the OpEval-2

traffic call sign procedure, as well as runway occupancy awareness scenarios. These low approaches

included four GA (Beech Bonanza, Cessna 210, Piper

Lance, Piper Aztec), two Beech King Air (A90, C90),

and two FAA Convair 580 aircraft.

Flight Period 2

Operations during Flight Period 2 were similar to

those of Flight Period 1, and they also were conducted

on RWY 17R. Flight crews conducted two full-stops,

one after each of the long approach spacing profiles in

the first 90 minutes of flight time, followed by low

approaches for the remaining 90 minutes. Table 1

shows that the same aircraft participated in Flight

Periods 1 and 2. In addition, four UPS aircraft performed three full-stops, one after each of the long

approach spacing profiles in the first 90 minutes of

Flight Period 2. Also, four UPS aircraft performed

three long-approach spacing profiles to a full stop

landing in the first 90 minutes of Flight Period 2.

Flight Period 3

Operations during Flight Period 3 consisted of four

full-stop sequences to RWY 35L. Long- approach spacing profiles were flown and operations were limited to

the Outer Covey pattern. Table 1 shows that only the

UPS and Federal Express aircraft participated in Flight

Period 3.

Flight Period 4

Operations during Flight Period 4 consisted of four

full-stop sequences to RWY 35L. Short- approach spacing profiles (i.e., closing to 3 miles or 90 seconds at the

runway threshold) were flown and operations again

were limited to the Outer Covey pattern. Table 1

shows that the UPS and Federal Express aircraft

participated in Flight Period 4, along with the L3 and

Collins business jets.

Flight Period 5

Flight Period 5 operations were similar to those of

Flight Period 2, and they were conducted on RWY

35L. Whereas in Flight Period 2, flight crews conducted long-approach spacing profiles; for Flight

Period 5, they conducted two full-stop, short-approach spacing profiles in the first 90 minutes of

flight time, followed by low approaches for the remaining 90 minutes. Table 1 shows that the same

aircraft participated in Flight Periods 2 and 5.

OPEVAL-2 APPLICATIONS

The five flight periods provided a means for evaluating each of the OpEval-2 applications. The following paragraphs briefly describe tasks and operations

that were developed and then embedded within the

flight periods to allow for evaluation of each OpEval2 application.

Airport Surface Situation Awareness (ASSA)

A set of circuitous taxi routes was designed to

evaluate the ability of flight crews to accurately navigate with either an electronic surface map display or

a paper surface map. The routes were created to assess

the extent to which a CDTI system that displayed

airport surface information as well as traffic could

improve the situation awareness of flight crews during

airport surface operations. This evaluation assessed

the flight crews’ ability to identify specific geographic

locations on the airport surface and the relative location of other aircraft and surface vehicles. Routes were

predefined, named, and provided to the controllers in

graphical format and to the flight crews in written text

format. Within each flight period, the taxi routes were

assigned to pre-selected aircraft on either their outbound or inbound taxi legs. The Ground controllers

cleared aircraft to taxi via the route name, and then

they monitored aircraft movements to ensure compliance with the prescribed route. Flight crews interpreted the written text as a series of turn instructions

to determine the route to taxi. Flight crews were asked

to stop the aircraft at a predetermined point along the

prescribed route to answer two questions posed by

observers. The first question asked them to identify

their current location and the second question asked

them to identify the most proximal surface traffic.

Additional information was recorded during normal

taxi to and from the ramp area.
4

Departure Spacing

Aircraft participating in the Outside Covey participated in the Departure Spacing application, which

was designed to evaluate the ability of the flight crew

and controllers to manage a pre-determined spacing

interval between departing aircraft. Long spacing (i.e.,

6.0 nm) and short spacing (i.e., 4.5 nm) scenarios

were defined for each flight period. Each time the

Outer Covey pattern was flown, flight crews determined the departure spacing interval between all but

the last aircraft, for which the local tower controller

provided the initial departure spacing interval.

Two procedures were used during OpEval-2 to

evaluate the Departure Spacing application and ensure compliance with standard ATC procedures. The

first procedure, known as flight crew-managed departure spacing, required flight crews to use mission

reference cards that outlined the procedures to follow

and distances to maintain. As part of this procedure,

controllers referred to scenario cards that defined the

departure spacing interval. Local control provided a

take-off clearance using standard phraseology when

standard departure separation was achieved from a

preceding departure (i.e., 6000 ft. and airborne). The

flight crews then positioned the aircraft on the OpEval2 runway, waited until the appropriate distance was

achieved, and then began the take-off roll. ATC

ensured the runway was protected while the flight

crew was waiting for the preceding departure to achieve

the appropriate distance.

In the second procedure, known as controllermanaged departure spacing, flight crews were required to use the mission reference cards and

controllers used the scenario cards that defined the tobe-achieved departure spacing interval. However, the

controller first placed the aircraft on the runway with

instructions to taxi into position and hold. The controller issued a take-off clearance using standard phraseology once the aircraft achieved the appropriate

departure spacing distance.

Initial and Final Approach Spacing

The purpose of the Initial and Final Approach

Spacing application was to evaluate the ability of the

flight crew and controllers to manage a pre-determined approach spacing interval between arriving

aircraft. The required spacing changed as the aircraft

continued along the pattern and approached the airport. The final spacing between aircraft as the lead

aircraft crossed the threshold was either 5 nm/120 sec

or 3 nm/90 sec (depending on the flight period). The

scenarios were designed to evaluate two levels of

CDTI approach spacing capability - an initial capability (herein referred to as UPS AT Build 2/4) and a

more advanced prototype avionics capability that

included approach spacing algorithms (herein referred to as Advanced Prototype). The traffic flows

from the Inner and Outer Covey patterns were merged

and sequenced by ATC at 20 miles from the runway

threshold. The initial approach spacing at the 20-mile

point was intended to be 6.5 nm (long spacing)

during some of the flight periods and 5.5 nm (short

spacing) during other flight periods. The ability of

ATC to meet these distances depended on several

factors, including spacing at the end of the departure

profile and separation from non-participating aircraft

that were inbound to the non-OpEval-2 runway.

Flight crews used CDTI during the 20-mile approach

to reduce the spacing interval to a long (5 nm or 120

sec if initial spacing was 6.5 nm) or a short (3 nm or

90 sec if initial spacing was 5.5 nm) final approach

spacing criteria. ATC monitored both participating

and non-participating aircraft and responded appropriately to ensure that at least minimum standard

separation was maintained.

Final Approach and Runway Occupancy

Awareness (FAROA)

The FAROA application included use of the CDTI

during OpEval-2 departure and arrival operations.

Flight crews were required to use CDTI immediately

prior to departures and arrivals to scan for targets that

were positioned on or near the runway. This procedure was used to determine if CDTI use increased

flight crew awareness of runway occupancy. Occasionally, an ADS-B equipped surface vehicle and/or

OpEval-2 aircraft were required to hold on the taxiways to provide flight crews using CDTI with targets

that were off the runway. Three flight periods included scenarios in which the FAA B727 and an ADSB equipped surface vehicle were positioned on the

OpEval-2 runway. When a vehicle was on the runway,

flight crews were cleared to complete a low approach

at 500 feet above ground level (AGL) and data were

collected on whether the flight crews became aware of

the conflict using CDTI and associated flight crew

workload.

Visual Acquisition and Traffic Awareness

The Visual Acquisition and Traffic Awareness application involved the use of ATC traffic advisories

during each of the five flight periods. The availability

of aircraft call sign on CDTI made it possible to

collect data on the use of aircraft call sign in traffic
5

advisories. Although this is not standard ATC phraseology, FAA Order 7110.65 does allow additional

information to be appended to traffic advisories.

Therefore, the flight profiles were designed to evaluate the inclusion of aircraft call sign in traffic advisories and replies by flight crews. Normal traffic calls

were used during the Departure and Approach Spacing applications for aircraft in the Outer Covey pattern. However, aircraft in the Inner Covey pattern

were vectored by ATC to create as many traffic calls as

possible during flight periods 1, 2 and 5. This procedure was used to assess the effect of including aircraft

call sign in traffic advisories when air traffic was

increased.

DESCRIPTION OF CDTI FUNCTION

The CDTI displays the relative position of proximal traffic to own aircraft. Additional information

also may be displayed, such as navigational aids and

obstructions. The CDTI can obtain traffic information from several different sources, including ADS-B,

Traffic Information Service (TIS), or an on-board

Traffic Alert and Collision Avoidance System (TCAS).

Traffic information is displayed via a dedicated avionics device (i.e., a CDTI), or the information can be

displayed on a shared avionics device (i.e., multifunction display). The latter form of display attempts

to integrate information from several sources into a

single display.

Figure 1 shows the three different CDTI platforms

that were evaluated during OpEval-2. These were the

UPS AT Build 2/4, the UPS AT MX20, and Prototype

Avionics. The UPS AT Build 2/4 platform included

two unique UPS AT Link and Display Processor Unit

(LDPU) versions that were identified as Build 2 and

Build 4. The Build 2 platform consisted of a Mode S

Transponder, TIS, and a ADC/Universal CDTI and

Keyboard. The Build 4 platform consisted of both a

Mode S and a UAT Transponder, and an Astronautics

CDTI and Keyboard. The Build 4 platform was a

newer (updated) version of the Build 2 platform. The

UPS AT MX-20 platform included an Apollo GX50

GPS receiver, a UAT transceiver, and the Apollo MX-20

multi-function cockpit display. Finally, the Prototype

Avionics platform consisted of an Advanced Prototype

capability and a Basic Prototype capability. The former

capability included either a surface map overlay and/or

an algorithm for advanced approach spacing. The Basic

Prototype included rudimentary CDTI capability and

was not intended to be representative of potential CDTI

implementation.

FLIGHTDECK OBSERVATION METHOD

Flight crew interaction with each of the CDTI

platforms was evaluated during the ASSA, Departure

Spacing, Visual Acquisition and Traffic Awareness,

Initial and Final Approach Spacing, and FAROA

OpEval-2 applications. As Table 1 illustrates, some

aircraft were not equipped to perform certain OpEval2 applications; hence, flightdeck observer data was

not collected for these applications. Flightdeck observer data aided in the identification of human factors

and performance issues associated with the use of each

CDTI display platform for the OpEval-2 applications.

Two methods of flightdeck observer data collection were designed to capture flight crew interaction

with CDTI platforms. The first method consisted of

flightdeck observation using trained observers. This

method allowed for direct observation of flight crew

procedures and recording of events by a flightdeck

observer. The second method entailed indirect observation and included the use of structured interviews

and a questionnaire. The structured interview was

completed by flight crews immediately after their

Figure 1. Illustration of the three different CDTI Platforms that were evaluated in OpEval-2. 
6

participation in each flight period. The questionnaire

was completed immediately after flight crews participated in their final OpEval-2 flight period.

Flightdeck Observers and Training

Nineteen flightdeck observers participated in

OpEval-2; all had previous flightdeck experience,

either as flight crews or as observers. Prior to OpEval2, the observers participated in a training program

that consisted of conference calls, email distribution

of materials, attendance at pre-OpEval-2 simulations,

and an eight-hour, on-site orientation session the day

before OpEval-2 began. The on-site orientation session provided a review of CDTI platform operating

procedures, and the final briefing on the flight schedule. Observers were trained in the following areas:

• Overview and use of the data-collection materials

• Normal flight crew operations procedures, including

use of checklists, typical workload management procedures, and sterile cockpit requirements

• Structured interview techniques

• OpEval-2 aircraft and flight assignments

• Review of the structure and timing of the OpEval-2

flight periods to which they were assigned.

The majority of flightdeck observer data were collected by direct observation of flight crew procedures

by an observer occupying the jump seat of the transport category aircraft, or one of the rear seats of the

single-pilot, general aviation aircraft. The single pilot

aircraft also carried a safety pilot occupying the right

seat. All flight crews were instructed to perform their

normal duties without regard to the presence of the

observer, and to freely exercise their judgment to

terminate data collection if in-flight circumstances

required it. Flight crews also were asked to verbalize

their thought processes so that the observers could

gain a better understanding of how the crews used

CDTI.

Observers were instructed not to interact directly

with crewmembers, except as required to collect data

for the ASSA and Initial and Final Approach Spacing

applications. In the instances where flight crew-observer interaction was necessary, data collection occurred only during ground operations with the aircraft

stopped and parking brakes set. Although observers

were asked to make all other observations silently as

the flight crew performed their normal cockpit duties,

the observer was considered a member of the flight

crew in all other respects and was expected to call

attention to any safety situation requiring the attention of the flight crew.

Flightdeck Observer Forms

Flightdeck observers recorded their observations

on the following forms, which were designed for

directly observing OpEval-2 applications and procedures during flight. A brief description of each form

is given in the following section and the actual forms

used in OpEval-2 are included in Appendix B:

• Checklists Form

• Visual Acquisition and Traffic Awareness Form

• Procedures Forms

• Modified Cooper-Harper Workload Scale

• Observers also administered the following materials

to flight crews at the end of OpEval-2 flight periods:

• Post-Flight Structured Interview

• Flight Crew Questionnaires

Checklists Form

This Checklists Form was used to determine the

effects, if any, of CDTI platforms on flight crew use

of checklists. Observers recorded whether checklists

were completed, not completed, suspended, interrupted, missed by the crew, etc. A frequency analysis

was used to analyze this data and identify the most

frequent causes of checklist intrusions. Written comments from the forms were used to illustrate examples

of specific types of intrusions.

Visual Acquisition and Traffic Awareness Form

This form was used to record how flight crews used

CDTI to aid in visual acquisition of traffic in the

Inner and Outer Covey patterns. The analysis of this

data focused on the order in which information was

accessed to support target acquisition during OpEval2 flight periods. In particular, the analysis determined

the extent to which the introduction of CDTI affected the procedure typically used to acquire targets

visually. A frequency analysis was used to assess the

effects of displays on the order in which information

was accessed to support target acquisition. Observers’

written comments were used to supplement the frequency analysis.

Procedures Forms

Two OpEval-2 Procedures Forms were used by

observers: the CDTI Build 2/4 Procedures Form and

the MX-20 Procedures Form. No forms were developed for the Prototype Avionics platform because its

development was in an early stage. Observers used the

Procedures Forms to assess crew use of recommended

CDTI techniques as defined in flight crew maneuver

cards. Analysis of data from the forms determined the

frequency with which crews used procedures defined

in the maneuver cards or deviated from these
7

procedures and created their own procedures. Deviations from suggested procedures may indicate flight

crew preferences or strategies for completing a procedure and these may be more efficient or “better” than

the strategies defined in the maneuver cards. From a

certification perspective, such preferences or strategies may represent acceptable and safe practices for

use of CDTI for OpEval-2 applications. Written

comments from this form were used to describe specific deviations in procedures that occurred frequently.

Modified Cooper-Harper Workload Scale

The Modified Cooper-Harper Workload Scale was

used by observers to assess flight crew workload of the

Pilot Flying (PF) and Pilot Not Flying (PNF) immediately after the crew used CDTI to perform the final

approach spacing application. The flight crew used a

scale that ranged from 1 to 10 to assign a workload

rating. Each number corresponded with a qualitative

description of the workload rating. For example, a

rating of 1 corresponded with “Mental effort is minimal and desired CDTI performance is easily attainable” whereas a rating of 10 corresponded with

“Acceptable CDTI performance cannot be accomplished reliably”. The ratings provided by flightcrew

members were analyzed for possible differences due to

several final approach spacing factors (e.g., time of

day, spacing interval).

Post-Flight Structured Interview

Following each OpEval-2 flight period, flightdeck

observers debriefed the participating flight crews by

conducting a structured interview. The interview

consisted of nine (MX-20) or ten (Build 2/4 and

Prototype Avionics) questions and was audiotaped.

Interview questions focused on procedures associated

with CDTI/MX-20 platform, training and preparation, interaction with ATC, and any tradeoffs that

may have been associated with the use of CDTI for the

OpEval-2 applications. Flight crew members were

interviewed together to promote discussion and elicit

additional information. After each interview, observers created a written summary of the audiotaped

recording that was used in the data analysis. The flight

crew responses to the questions in the structured

interview form were categorized and aggregated with

the flight crew responses to the open-ended questions

from the respective flight crew questionnaires to identify operational and human factors issues associated

with use of CDTI for OpEval-2 applications.

Flight Crew Questionnaires

CDTI Build 2/4 and Prototype Platforms

A 173-item questionnaire was the primary means

for gathering information on flight crew demographics, previous CDTI experience and training, and

acceptability of the following characteristics associated with OpEval-2 applications:

• CDTI functionality

• Use of color in display modes

• Display symbology

• Display features

• Display location and readability

Descriptive statistics are reported for the questionnaire data, and factor analyses were used to determine if

particular CDTI features or functional components

could be grouped according to some underlying factor.

Responses to open-ended questions were analyzed to

identify other issues that the flight crew judged to be

pertinent. Open-ended responses from this questionnaire were aggregated with responses from the structured

interviews to summarize operational and human factors

issues associated with OpEval-2 applications.

MX-20 Platform

A 163-item questionnaire served as the primary

means for gathering information on crew demographics, previous MX-20 experience and training, and

acceptability of the following characteristics associated with OpEval-2 applications:

• CDTI functionality

• Use of color in display modes

• Display symbology

• Display features

• Display location and readability

The analysis of this questionnaire was similar to

that used to analyze the Build 2/4 questionnaire.

Data Analysis

All data were screened for errors, outliers, and other

anomalies that would bias statistical analyses. Descriptive statistics and frequency analyses were computed

where appropriate, as were ANOVA and principal components analysis. The following section provides a summary of results that are based on analyses of data from

each of the flightdeck observer forms. OpEval-2 applications and effects associated with CDTI platforms also

are reported in this section.
8

RESULTS

Airport Surface Situation Awareness

A set of circuitous taxi routes was designed to evaluate

the ability of flight crews to accurately navigate with

either an electronic surface map or a paper surface map.

The routes were created to assess the extent to which the

CDTI platform could be used for the ASSA application

to improve the situation awareness of flight crews during

airport surface operations. This evaluation assessed flight

crews’ ability to identify specific geographic locations on

the airport surface and the relative location of other

aircraft and surface vehicles. Routes were predefined,

named, and provided to the controllers in graphical and

text format and to the flight crews in written text format.

Figure 2 shows that the flightdeck observers noted

that 53% of flight crews used CDTI to identify

Traffic to Follow (TTF) during taxi operations. Of

this group, a larger percentage of flight crews used an

electronic surface map (65% vs. 40% with a paper

surface map). Additional analyses revealed that the

flightdeck observers reported that flight crews used

CDTI for enhancing surface traffic awareness 63% of

the time. Of these positive responses, 82% of flight

crews used an electronic surface map, and 40% had a

paper surface map (Figure 4). Flightdeck observers

also noted that 53% of flight crews used CDTI to

assess the location of traffic on the airport surface. Of

this group, 76% used an electronic surface map, and

27% used a paper surface map (Figure 4).

Figure 2. Use of CDTI to locate Traffic-to-Follow (TTF) in taxi sequence by flight crews 

with an electronic surface map and those with a paper surface map.

0

20

40

60

80

100

Yes No Missing Data

Used CDTI to Locate Traffic to Follow

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Surface Map (n=17) No Surface Map (n=15)

Figure 3. Use of CDTI for surface traffic awareness by flight crews with an electronic 

surface map and those with a paper surface map. 

0

20

40

60

80

100

Yes No Missing Data

Used CDTI for Surface Traffic Awareness

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Surface Map (n=17) No Surface Map (n=15)
9

Flightdeck observers also recorded flight crew use

of two CDTI platform features: the Target Selection

function and the Ground Track Vector function.

Overall, 31% of flight crews used the Target Selection

function on CDTI to identify TTF during taxi operations. Approximately 12% of flight crews used the

Ground Track Vector function on CDTI to evaluate

the target speed during airport surface operations.

These percentages were nearly equivalent for flight

crews with an electronic surface map and those with

a paper surface map.

Flightdeck observer data indicated that one-third

of the flight crews utilized CDTI to identify taxiway

locations on the airport surface. Of this group, 65%

of flight crews with an electronic surface map used the

CDTI to identify taxiway locations during surface

navigation. None of the flight crews with a paper

surface map used CDTI for this purpose.

Flightdeck observers also recorded whether flight

crews correctly identified the most proximal surface

traffic when they were asked this question. The data

indicated that only 12 of the 32 flight crews responded to this question. Of the 12 flight crews, 7 had

the electronic surface map, and all correctly identified

the most proximal traffic. Four of the five flight crews

with a paper surface map correctly identified the most

proximal traffic.

FAROA

Flight crews were instructed to utilize the CDTI to

assess runway occupancy prior to entering or crossing

runways (FAROA ground). During airborne operations from the final approach fix through rollout,

flight crews were instructed to utilize CDTI to

continuously assess runway occupancy (FAROA air).

Flightdeck observers recorded data for the FAROA

application during these airborne and ground events.

Additionally, during Flight Periods 1 and 2, a surface

vehicle and an aircraft were positioned at two locations on the active runway to evaluate the effect of the

CDTI on flight crew awareness of runway occupancy.

FAROA Air

Flightdeck observer data indicated that during

FAROA Air operations 62% of flight crews used the

CDTI to evaluate the runway environment for possible conflicts. Figure 5 shows that within this group,

71% of flight crews used an electronic surface map,

and 56% used a paper surface map. During final on

low approaches, flightdeck observers reported that

flight crews using CDTI equipped with an electronic

surface map were more effective at determining when

the runway was occupied than flight crews with the

paper surface map. Eight flight crews with an electronic surface map identified the traffic on the runway, whereas flight crews with a paper surface map

failed to identify the traffic on the runway. These

results suggest that the addition of an electronic surface

map may aid in identifying traffic on a runway.

Flightdeck observers also recorded CDTI display

modes that were used most frequently by flight crews

during the FAROA Air application. Observer data indicated Arc mode was used 51% of the time by the flight

crews who had a paper surface map. The Compass Rose

and No Arc/No Rose modes both were used nearly 21%

of the time. The Compass Rose mode was selected 39%

of the time and the Arc mode 14% of the time by the

flight crews who had an electronic surface map.

Figure 4. Use of CDTI to assess traffic location on the airport surface by flight crews 

with an electronic surface map and those with a paper surface map. 

0

20

40

60

80

100

Yes No Missing Data

Used CDTI to Locate Traffic on Surface

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Surface Map (n=17) No Surface Map (n=15)
10

Flight crew use of two CDTI platform features, the

Range function and the Ground Track Vector function, was recorded by flightdeck observers during the

FAROA Air application. Approximately 53% of the

flight crews used the Range function to adjust viewing

range while on the final approach. Of this group, 69%

of flight crews with an electronic surface map utilized

the Range function, and 41% of flight crews with a

paper surface map used the function. Observer data

indicated that only 23% of flight crews used the

Ground Track Vector function to evaluate the target

speed of the leading aircraft or to identify an aircraft

accelerating onto the runway. These flight crews were

split evenly between those with an electronic surface

map and those with a paper surface map.

FAROA Ground

Flightdeck observers reported that during FAROA

Ground operations, 56% of flight crews used CDTI

to assess runway occupancy. Figure 6 reveals that

nearly 71% of flight crews with an electronic surface

map and 40% of crews with a paper surface map used

CDTI to support this task. A majority of flight crews

used CDTI to check for traffic on the approach, and

half of the crews used the CDTI to check the departure

corridors for possible conflicts. Again, flight crews

with an electronic surface map utilized CDTI with

greater frequency (71% for approach and 59% for

departure, respectively), than flight crews with a paper surface map (47% for approach and 40% departure, respectively).

Figure 5. Use of the CDTI during FAROA Air operations to evaluate runway environment for 

possible conflicts by flight crews with an electronic surface map and those with a paper 

surface map. 

0

20

40

60

80

100

Yes No Missing Data

Used CDTI to Evaluate Runway Environment

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Surface Map (n=49) No Surface Map (n=67)

Figure 6. Use of the CDTI during FARAO Ground operations for runway occupancy awareness 

by flight crews with an electronic surface map and those with a paper surface map. 

0

20

40

60

80

100

Yes No Missing Data

Used CDTI for Runway Occupancy Awareness

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Surface Map (n=17) No Surface Map (n=15)
11

Flightdeck observers noted that 65% of flight crews

used the Range function. Flight crews with a surface

map tended to use the Range function more often

(71%) than those with no surface map (53%). Only

12% of the crews with a surface map and none of the

crews without a surface map used the Ground Track

Vector function to evaluate target speed.

Checklists

Flightdeck observers recorded flight crew behavior

associated with checklists used during the flight. Data

from the Checklist Observer Form were used to determine the effects of the four display types on flight

crew use of checklists. Observers recorded whether

checklists were Completed, Suspended, Interrupted,

Not Completed, Missed by the Crew, or Missed by

the Observer. In addition, observers were told to use

Other as a category, if necessary. Preliminary analyses

showed very low counts in the Suspended, Interrupted, Not Completed, Missed by the Crew, and

Missed by the Observer categories. These categories

were later pooled into two more general categories:

Not Completed, which included the original Not

Completed and Missed by Crew categories; and, Completed, which the included the original Completed,

Suspended, and Interrupted categories.

A frequency analysis was used to determine checklist completion rates across the four different traffic

display types. After recoding the data into the Completed and Not Completed categories, the data were

statistically analyzed using a chi-square test. The

results of the frequency analysis are shown in Figure

7, which shows checklist completion rates by display

type. Overall, the checklist completion rate was equivalent across display types, and it was greater than 98%

for all display types. Completion rates were not affected by time of day (i.e., day vs. night) in which the

OpEval-2 flight periods occurred.

Although use of CDTI did not appear to affect

checklist completion rates, written comments by some

of the flight crews using each display type suggested

that the timely completion of checklists came with a

cost (viz., additional workload). For example, interpretation of the flight crew data suggested that two of

the five checklist suspensions and five of the ten

checklist interruptions were due to CDTI. Another

factor that may have induced workload was the closed

traffic pattern operations used in OpEval-2, which

required repetitive checklist operations that are not

typical in normal flight operations, thereby increasing

workload. The checklist data do not reveal whether

the increase in perceived workload was due to the

novel displays, per se, the higher-than-normal checklist repetition resulting from the closed traffic patterns used in OpEval-2, or the interaction of these two

factors. Some flight crews reported that their perceived workload was reduced as they became more

familiar with the displays.

Visual Acquisition and Traffic Awareness Form

Observers used the Visual Acquisition and Traffic

Awareness Form to record the order and manner in

which the flight crews used three different information

methods to enhance traffic awareness while in the Outer

Figure 7. Checklist completion rate by platform type. 

0%

20%

40%

60%

80%

100%

CDTI Bld 2/4 Basic Prototype Advanced

Prototype

MX20/GX60

Equipment Type

Completed

Not Completed
12

and Inner Covey traffic patterns. These three methods

included use of the CDTI, ATC traffic advisories, and

visual acquisition of traffic.

The analysis of the acquisition data focused on the

order in which information was accessed to support

target acquisition and traffic awareness during OpEval2 flight periods. In particular, the analysis sought to

determine the extent to which the introduction of the

display platform affected the procedure typically used to

acquire targets (i.e., ATC advisory/visual acquisition),

during both the night and the day flight periods. A

frequency analysis, which included a chi-square test, was

used to determine if the three methods were used equally

often. The frequency analysis was also used to define the

order in which methods were used to support target

acquisition and traffic awareness.

The chi-square test for the first method used (CDTI)

was significant, χ2

(2)=17.49, p<.0001. During day and

night events, the observed usage rate for the CDTI

method was significantly higher than expected, whereas

the observed usage for the visual method was significantly lower than expected. As Figures 8 and 9 show,

flight crews initially identified nearly two-thirds of the

traffic targets using the traffic display. The chi-square

test for the second (ATC Traffic Advisors), χ2

 (2)=10.94,

p<.004, and third methods (Visual Acquisition), χ2

(2)=10.94, p<.004, used also were significant.

Interestingly, Figure 8 reveals that flight crews were

somewhat more likely to use ATC traffic advisories than

visual sighting as the second method during day operations. However, during night operations the flight crews

were far more likely to use visual sighting than ATC

traffic advisories as the second method. This effect is

attributable to the better visibility that prevailed during

OpEval-2 night flight periods. The traffic awareness

benefits associated with the CDTI were more apparent

during day flight periods when the visibility was poor

(hazy VFR). Overall, about 75% of all traffic events

involved use of the CDTI.

Modified Cooper-Harper Workload Scale

Flightdeck observers used the Modified CooperHarper Workload Scale to obtain a workload rating

from the Pilot Flying (PF) immediately after the flight

crew used the platform to perform the Final Approach

Spacing application. After each full-stop landing, flight

crews using CDTI Build 2/4, Advanced Prototype, and

Basic Prototype platforms rated perceived workload

using the scale items, which were read by the observer

after the flight crew set the aircraft brakes. Because flight

crew tasks were not identical and comparisons between

platform types based on workload ratings alone would be

misleading, the workload ratings were analyzed within

platform types for possible differences due to time of

day (i.e., day vs. night) and final approach spacing

interval (closing to 5 nm vs. 3 nm). No significant

differences were found for time of day or final approach spacing interval.

Pilot workload ratings for the Final Approach

Spacing application varied by CDTI platform. Pilot

workload ratings are shown in Table 2. The rating 2.9

corresponds to the Modified Cooper-Harper scale

value “Mental effort is required to attain acceptable

performance.” The rating 8.2 corresponds to the

Figure 8. Three methods used for detecting traffic and the order in which flight 

crews used these methods during OpEval-2 day flight periods. 

0%

20%

40%

60%

80%

100%

Visual Display ATC

Traffic Acquisition Type

First

Second

Third

Order of 

Acquisition
13

Modified Cooper-Harper scale value “Maximum

mental effort is required to avoid large or numerous

errors.” The rating 4.8 corresponds to the Modified

Cooper-Harper scale value “ High mental effort is

required to attain acceptable performance.”

Flight crew tasks for the Final Approach Spacing

application varied as a function of platform type. Flight

crews using CDTI Build 2/4 and the Basic Prototype

platforms were to achieve a desired spacing at the threshold by observing the raw speed, closure and range data on

CDTI and determining the necessary speed adjustments

to achieve the desired spacing behind TTF. Flight crews

using the Advanced Prototype were to adjust speed in

response to a speed command, which was computed as

a function of planned approach speeds, current speeds

and distance to threshold. This speed command was

displayed as a supplementary cursor on the EFIS airspeed display. For these flight crews, matching current

speed with commanded speed should have resulted in

the targets spacing in seconds behind TTF. Flight crews

using MX-20 did not perform spacing tasks.

None of the displays had been certified for use in

the Final Approach Spacing application and only the

Advanced Prototype platform had design features

that specifically supported the spacing-at-threshold

task. Nevertheless, flight crews using CDTI Build 2/

4 platform rated the workload lower than the flight

crews using the other two platform types. Flight crews

using the Basic Prototype rated the workload highest,

and flight crews using the Advanced Prototype display rated workload as intermediate (i.e., lower than

Basic Prototype platform, but higher than CDTI

Build 2/4 platform).

Anecdotal evidence from the observers suggests

that in some cases, the PF may have rated both the

Initial Approach Spacing and Final Approach Spacing tasks and not just the latter task. Workload ratings

should be interpreted with caution due to the differences between crew tasks, platform capability and

small sample sizes for some ratings.

Figure 9. Three methods used for detecting traffic and the order in which flight 

crews used these methods during OpEval-2 night flight periods. 

0%

20%

40%

60%

80%

100%

Visual Display ATC

Traffic Acquisition Type

First

Second

Third

Order of 

Acquisition

Table 2. Mean pilot workload ratings as a function of CDTI platform.

Platform Type N Mean Rating Standard Deviation 

CDTI Build 2/4 42 2.9 2.2 

Basic Prototype 21 8.2 1.5 

Advanced Prototype 15 4.8 0.9 
14

Procedures Form

CDTI Build 2/4 Platform

The CDTI Build 2/4 platform flight crew Mission

Cards specified tasks to be accomplished using CDTI

information. For the Departure Spacing application,

the task was to determine the assigned separation

interval (either 4.5 nm or 6.0 nm) between ownship

and TTF on departure, and then release brakes for

takeoff. A completion standard was not specified in

the Mission Card for this task, and flightdeck observers used their own judgment to determine whether the

task was completed.

The Initial Approach Spacing application task required flight crews to establish and maintain the

target separation interval on the downwind leg as

defined for that flight period. The target value was

either 7 nm or 9 nm, with a completion standard of ±

1 nm. The Final Approach Spacing application task

required flight crews to close to the target separation

distance at the runway threshold. Target values were

either 5 nm or 3 nm, with a completion standard of

+1.0 nm to 0.5 nm. Task completion rates for flight

crews using the CDTI Build 2/4 platform are presented in Table 3.

Completion rates were approximately the same for

the three OpEval-2 applications. These data provide

some insight into the ability of flight crews to follow

a procedure for achieving an assigned spacing based

on the use of CDTI. The results may be useful for

discussing minimum acceptable reliability of performance in an actual operational setting when ATC may

be relying on a flight crew’s ability to achieve an

assigned spacing.

The flight crew Mission Cards also provided flight

crews using the CDTI Build 2/4 platform with suggested techniques for completing the applications.

Techniques that related specifically to the use of

CDTI information were evaluated by the flightdeck

observers in accordance with Table 4. These techniques were included in the Procedures Form, which

was used to record the data for each circuit flown by

aircraft in the Outer Covey pattern. These data were

Table 3. CDTI Build 2/4 platform task completion by OpEval-2 application (in percent). 

Task Completed Departure 

Spacing 

Initial Approach 

Spacing 

Final Approach 

Spacing 

Yes 83.3 86.7 88.5 

No 10.0 8.9 11.5 

Missing Data 6.7 4.4 0.0 

Table 4. CDTI Build 2/4 platform tasks and recommended techniques for Departure Spacing, Initial 

Approach Spacing and Final Approach Spacing applications.

Departure 

Spacing 

Initial 

Approach 

Spacing 

Final Approach 

Spacing 

Recommended Techniques for OpEval-2 

Applications Determine 

departure 

separation 

interval 

Establish and 

maintain desired 

initial separation 

distance 

Establish and 

maintain desired 

final approach 

separation 

Set range ring to separation distance (SET_RR) ✓ ✓ ✓

Adjust display range prior to takeoff 

(RUP_RDN) ✓ 

Set the desired altitude range for traffic display 

for the application (USE_LVL) ✓ ✓

Use vector display to evaluate target speed 

(USE_VEC) ✓ ✓

Select TGT (SEL) to identify target (SEL_TGT) ✓ ✓

Set vector time to 2 min Long or 1.5 min Short 

(SET_VEC) ✓ ✓ ✓

Eval spacing by PNF only ✓
15

aggregated across circuits to compute overall percentage of use. Percentages for each technique are presented by OpEval-2 application.

Departure Spacing

Data on the use of each of the techniques during

the Departure Spacing application are provided in

Figure 10. The most notable deviation from the

recommended techniques was in the use of the altitude range filter (USE_LVL). This display feature,

which can be adjusted by the flight crew, determines

the altitude range in which ADS-B targets would be

displayed. The default value displays targets up to

2500 feet above and below ownship. In LK-mode,

which was the recommended departure setting, the

range above ownship changes to 9500 feet (the system

default value), or a value set by the flight crew. The

purpose of the recommendation is to ensure that the

display continuously shows the target of interest during its climb-out on departure. Most flight crews did

not change the setting prior to departure. The flight

crew Mission Card for the preceding application (i.e.,

FAROA Ground) suggested that this mode be selected; thus, further adjustment was not necessary. In

addition, flightdeck observers reported that the climb

performance of departing aircraft during OpEval-2

was such that the aircraft did not climb above the

altitude filter setting (and thus disappear from the

display) regardless of its setting. Therefore, even when

they did not set CDTI display features as suggested,

flight crews were not prompted to adjust settings to

the recommended configuration because the departing aircraft did not disappear from the display. Finally, observers noted that in most cases, settings

made at the ramp before departure were often not

changed for the duration of the flight.

In most aircraft, the Range Ring (SET_RR) was set

to a value appropriate for tasks in the Departure

Spacing application. Fewer flight crews were seen

adjusting or heard verbally verifying that the display

range was set prior to departure (RUP_RDN). For the

short-spacing departure interval (closing to 4.5 nm),

the best display resolution was at the five-mile scale;

however, a longer scale also could be used, and some

flight crews may have decided that an adjustment was

unnecessary under the circumstances. In fact, there

would have been no loss of accuracy from using the

longer display range if flight crews had highlighted

the target using the SEL function. Target Selection

behavior for tasks in the Departure Spacing application was not assessed specifically, nor did observers

record their own observations regarding Target Selection. However, informal discussions with observers

indicate that many flight crews retained the Target

Selection that they used for tasks in the ASSA and

FAROA Ground applications, which preceded Departure Spacing and occurred during taxi operations.

Once selected, the range to the target to the nearest

tenth of a mile would be available in its displayed data

tag regardless of display range setting or range ring

value. In lieu of the range ring, this information could

be used readily to support tasks in the Departure

Spacing application. Direct alphanumeric readout of

target range using the Select function would provide

the most accurate data given the size and resolution of

the display, and the size of the traffic symbology.

Similarly, although most flight crews adjusted vector display time to the recommended value

(SET_VEC), some did not. The recommendation to

set the vector time supported the Initial Approach

Spacing task, and not the immediate tasks in the

Departure Spacing application. Flight crews who

Figure 10. Flight crew techniques using CDTI Build2/4 platform for the 

Departure Spacing application. 

0

20

40

60

80

100

SET_RR RUP_RDN SET_VEC USE_LVL

Techniques Used

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Yes No Missing Data
16

elected not to set the display may have correctly

determined that it did not provide useful information

for the tasks in the Departure Spacing application.

The vector display is time-based, providing an indication of ownship position at a selectable future time,

and the tasks in the Departure Spacing application

were distance-based. This particular technique also

had the highest missed data rate; some observers notes

recorded difficulty in determining the status of the

display from their vantage point in the cockpit

jumpseat.

Initial Approach Spacing

Data on the use of techniques during the Initial

Approach Spacing application appear in Figure 11.

The notable deviations from the recommended techniques during tasks associated with the Initial Approach Spacing application were in the use of the

Altitude Filter (SET_TGT) and use of the Vector

Time Display (USE_VEC) to evaluate target speed.

The lack of observed compliance with these techniques is due in part to a difference between the

techniques suggested on the Mission Card and the

techniques listed on the Procedures Form. Adjustment of the altitude filter and use of the vector display

to evaluate target speed are not among the techniques

recommended on the Mission Card. Therefore, it is

not surprising that observed compliance is low.

Flightdeck observer comments indicated that the altitude filter setting made on the ramp was sometimes

not changed for the duration of the flight. The relatively higher compliance with the SEL_TGT technique and several recorded observer comments

indicated that TTF speed information was most commonly acquired from the alphanumeric readout of

selected target speed. In addition, there were two

recorded observations of active use of the closure rate

display to capture the initial approach spacing interval.

The highest compliance rate for techniques in the

Initial Approach Spacing application was setting the

vector time (SET VEC) to recommended values.

However, several observer comments indicated that

high compliance rates were not due to active choices

made by the flight crews. Instead, the rates were

attributable to initial selections made by flight crews

while they were on the ramp or during previous

application tasks. No observer reported active use of

the vector display for this application. As was the case

for the Departure Spacing application, the Initial

Approach Spacing application was fundamentally a

distance-based, in-trail separation task. In contrast,

the ownship vector is time-based, and thus would

represent the correct spacing only at one particular

true airspeed.

Final Approach Spacing

For the Final Approach Spacing Application, the

Build 2/4 flight crew Mission Card specified two of

the five techniques evaluated by flightdeck observers:

use of Target Selection (SEL_TGT) and evaluation of

spacing by the PNF only. The latter technique is not

a CDTI function; rather, it is a crew resource management technique for using CDTI. As shown in Figure

12, Target Selection was used on most of the arrivals,

whereas evaluation of spacing by PNF only was judged

by observers to have occurred just over half the time.

The Target Select function provides essential information for tasks associated with the Final Approach

Spacing application, including the ground speed of

the selected target and closure rate display. The high

Figure 11. Flight crew techniques using CDTI Build2/4 platform for the Departure 

Spacing application. 

0

20

40

60

80

100

SET_RR SET_VEC USE_LVL USE_VEC SEL_TGT

Techniques Used

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Yes No Missing Data
17

percentage of use of this feature may reflect the importance of such information. Flight crews reported

using this information as a primary source of information about the TTF during the Final Approach Spacing application, a finding confirmed by the flightdeck

observers.

The lower rate associated with the evaluation of

spacing by PNF is noteworthy, given that the intent of

the technique was to enable the PF to concentrate on

the aircraft control tasks associated with the final

approach phase of flight. When using this technique,

the PNF manipulates the traffic display and provides

the PF with appropriate information. Flightdeck observer data indicated that about a third of PFs devoted

some attention to the display during the final approach spacing task. One explanation for these results

may be that the novelty of CDTI captured the attention of the entire flight crew. Prior experience with the

implementation of the Traffic Alert and Collision

Avoidance System (TCAS) suggests that this novelty

phenomenon may be short-lived, diminishing with

additional experience. More effective flightdeck procedures based on a thorough task analysis of the target

procedures and emphasis during training also may

enhance performance.

Post-Flight Structured Interview Form

Flightdeck observers debriefed flight crews using a

structured interview. A content analysis of the resulting interview summaries was conducted to identify

the operational and human factors issues that

crewmembers found to be particularly relevant to the

OpEval-2 applications. Content analyses were performed on 35 interview summaries that represented

the responses of 38 unique crewmembers. Approximately 430 individual statements were evaluated and

categorized. Crewmember responses for all CDTI

platforms were represented. Flight crews using the

CDTI Build 2/4 platform composed nearly 53% of

the responses.

Interpretation of the interview data was complicated by factors related to the method used to collect

it. For example, interviewers were free to choose their

own method of transcription and subsequent reporting of the data. There was significant variation in the

level of detail provided for analysis, from very short,

written summaries to a verbatim transcript. In at least

one case, the interview questions appeared to have

been answered in writing, with no additional discussion by the interviewer and crewmembers. Given

these factors, it is possible that additional issues may

have been raised by the crew and not captured in the

written summaries. Furthermore, the relative proportion of responses within the issue categories shown in

Figure 13 may not be reliable because these proportions are based on a small number of crewmember

responses. Based on these factors, the structured interview data are reported in aggregate, with the intent

of providing a broad overview of the opinions and

concerns about operational and human factors issues

that were expressed by flight crews who participated

in OpEval-2.

• The content analysis suggested five broad categories

of issues:

• Applications - Direct comments on OpEval-2 applications.

• ATC Interaction - Reported effects of CDTI on the

ATC procedures, communications, or comments related to flightdeck/ATC procedures integration.

• Crew Resource Management (CRM) - Flight crew

interaction, use of checklists, flightdeck procedures

Figure 12. Flight crew techniques using CDTI Build2/4 platform for the Final 

Approach Spacing application. 

0

20

40

60

80

100

SET_RR SET_VEC USE_VEC SEL_TGT Eval by PNF

Techniques Used

F

r

e

q

u

e

n

c

y

 

(

i

n

 

%

)

Yes No Missing Data
18

design, workload, tradeoffs, and other aspects of

flightdeck operations.

• System - Comments related to the design and operation of the CDTI, or its operating context; includes,

controls, displays, installation and other aspects of

usability.

• Training - Platform-specific training concerns and

OpEval-2 learning effects.

The two largest categories are System and Crew

Resource Management (CRM), followed by Training, ATC Interaction, and Applications. Positive and

negative reactions were recorded within each category, but the general trend was to focus on system

deficiencies or issues that crews felt needed improvement. The responses within a category were separated

by CDTI platform type to determine if the platform

types were associated more frequently with particular

comments.

System

Mixed Equipage

Flight crews expressed concerns about the utility of

CDTI platforms in a mixed or partial equipage environment. Two ADS-B links were evaluated during

OpEval-2, and each was “blind” to the other. That is,

flight crews who used a CDTI platform with one of

the ADS-B links could not see or be seen by flight

crews using a CDTI platform with the other ADS-B

link. Of course, unequipped aircraft also would not be

seen on CDTI regardless of the ADS-B link used. A

large majority of crewmembers commented on the

enhanced situation awareness benefit that would be

derived if all aircraft were displayed. As would be

expected with CDTI use, some flight crews also noted

that they would have to continue their visual search

and “see-and-avoid,” out-the-window surveillance

procedure to detect and avoid unequipped aircraft in

a mixed-equipage environment. One of the flight

crews using the Advanced Prototype platform commented very favorably on its advanced TCAS/ADS-B

system, which displayed both ADS-B and transponder-equipped aircraft, essentially providing an approximation of the full equipage that crews thought

would be necessary to achieve full CDTI benefits.

Future operational evaluations include plans to demonstrate a multilateration-based Traffic Information

System - Broadcast (TIS-B) that would enable the

display of all transponder-equipped aircraft during

surface and near-surface observations.

Display Clutter, Display Resolution, Control of

Display Range

The second most frequently mentioned issue in the

System category was display clutter. In addition, there

were several related comments on display resolution,

which refers to the ability of the display to illustrate

targets in sufficient detail so that flight crews can

resolve them from other information. Display-range

control, which is one means of removing unnecessary

information from the display, also was mentioned by

flight crews. Examples of the effects of clutter mentioned by the crews included the momentary merging

of the TTF target with a grouping of surface targets as

TTF approached the runway. Flight crews reported

that this complicated tasks associated with the FAROA

Air application for the airplane that was following,

and made it more difficult to determine runway

occupancy. The runway occupancy task also highlighted display resolution problems, especially when

conducted without benefit of an electronic surface

map. Flight crews had to infer from their own runway

alignment and positions of other displayed aircraft

whether or not TTF was on the runway. The comments regarding control of display range also were

related to tasks associated with the FAROA Air application. These comments referenced both workload

issues and display size and resolution. The available

display scales and the size of the display required

manual adjustment of range to allow monitoring of

the runway environment. The most suitable scale

(generally agreed to be 1 or 2 miles) had to be selected

after passing the final approach fix and during the

final descent for landing, which is normally when

CRM

30%

(129)

ATC Interaction

11%

(48)

Applications

7%

(36)

Training

18%

(77)

System

34%

(140)

(Number of statements is in parentheses)

Figure 13. Percentage of interview statements by 

category for all CDTI platforms.
19

flight crews are devoting full attention to the landing

task. Several crews noted additional workload associated with reconfiguring the CDTI.

Overall, flight crew comments about clutter indicated that they desired a more effective means to

eliminate unnecessary information and configure the

CDTI to provide the information necessary to support OpEval-2 applications.

Display Integration

The last issue in the System category mentioned

frequently by flight crews was display integration.

Flight crews from the CDTI Build 2/4, Advanced

Prototype and Basic Prototype platforms all noted

that traffic integration with the flightdeck navigation

display would provide the most effective tactical picture for situation awareness. Additionally, flight crews

using the Advanced Prototype platform desired integration of the approach spacing speed commands

with the Primary Flight Display airspeed tape to

reduce the mental and physical workload of translating a stand-alone speed command into a speed control

response. Flight crews using the CDTI Build 2/4

platform became aware of the integration issue while

performing tasks associated with the Final Approach

Spacing application. They noted competition for

attention between the Head-Up Display (HUD),

which was providing primary flight control information, and the centrally located CDTI. This competition occurred despite a recommended technique that

suggested the PNF only attend to the CDTI. Perhaps

with practice flight crews may find it easier to disregard the display and rely on the PNF conveyance of

display information. Without the benefit of practice,

flight crews using the CDTI Build 2/4 platform

suggested that providing spacing data on the HUD

would be more effective than verbal transmission of

the information from the PNF.

Crew Resource Management

The second-most frequent set of issues mentioned

in the structured interview were related to Crew

Resource Management (CRM). This category included

comments on workload, procedures, allocation of PF

and PNF duties, intra-cockpit communication, and

situation awareness. Nearly 80% of CRM-related

comments from flight crews using the CDTI Build 2/

4 platform related to workload. These comments

included mention by flight crews of workload being

higher than that in normal line operations, distraction

from other tasks, and crew management issues related

to extracting information from the CDTI.

Workload and Task Interference

Flight crew comments about workload were usually in the context of the Final Approach Spacing

application. Crews reported that the level of perceived

workload depended somewhat on the distance that

needed to be made up during the final segment. The

technique of using the Pilot not flying (PNF) to

provide information from the display to the PF appeared to be effective in performance terms, but it

apparently increased workload, primarily for the PNF.

Effects on the PNF included a heavier checklist management burden when it was necessary to suspend an

ongoing checklist to provide spacing information,

and interference with the monitoring of airspeed and

altitude during the final approach segment inside the

final approach fix. Several flight crews using the Build

2/4 CDTI platform indicated that they missed some

of the ATC communications due to PNF workload.

Flight crews using the Advanced Prototype platform also expressed concern about a higher than

expected workload when using the advanced spacing

algorithm speed commands during the Final Approach Spacing application. However, these crews

indicated that this task had little impact on crew

interaction. These comments may be due to the integrated speed command, which placed a command bug

directly on the speed tape, eliminating the requirement for verbal transfer of the speed command data by

the PNF. The high reported workload for the Final

Approach Spacing application was related to the frequency and magnitude of speed adjustments. Under

certain conditions, flight crews judged the frequency of

adjustment to be too great, and the magnitude too large.

A crewmember using the Basic Prototype platform

noted that the scenarios within the OpEval-2 flight

periods induced a higher-than-normal workload due

to the closed traffic patterns and repetitive checklist

requirements. This compressed the normal arrival

workload into a much shorter time period, even

without the additional CDTI-related spacing tasks.

As noted by more than one flight crew, successful

integration of CDTI procedures with ongoing

flightdeck tasks will require good management skills

supported by well-designed and explicit procedures.

Interestingly, the structured interview data revealed

that flight crews using the MX-20 platform gave many

CRM-related comments; but, these indicated few

effects on workload and no tradeoffs between attending to the CDTI and other cockpit tasks. Several

factors could explain the different crew reactions

based on CDTI platform. Flight crews using the MX20 platform were operating as single-pilot and safety
20

pilot crews. However, the MX-20 pilots generally

were most familiar with single pilot operations and

were not given specific training in CRM, or how to

use their safety pilot resource efficiently. Therefore,

although CRM seemed to be a difficult task, it did not

seem to interfere with other cockpit operations. In

addition, pilots using the MX-20 platform also were

not assigned the spacing tasks that generated a significant proportion of workload comments from the

flight crews flying other CDTI platforms.

Finally, many flight crews using the MX-20 platform were flying personally owned aircraft and were

very familiar with the operation of their traffic displays. In contrast, many flight crews using other

CDTI platforms did not see the platform in operation

until the beginning of OpEval-2. They had minimal

training on the CDTI platform and no formal training on integrating CDTI procedures with OpEval-2

applications. The platform was being flown in “normal” flight operations, but the OpEval-2 flight periods were not equal to normal revenue operations

experienced by line flight crews. Hence, these data

provide a snapshot of the workload that existed during OpEval-2 conditions and further research and

analysis is required to adequately characterize the

workload associated with CDTI platforms when they

are used during normal revenue operations.

Situation Awareness

Enhanced situation awareness was reported by flight

crews using all CDTI platforms and was a consistently

positive finding in the interview statements. Among

the reported benefits were perceived improvements in

tasks associated with the Final Approach Spacing

application, improved awareness of local traffic situations, and the ability to determine the best merge

into an existing traffic flow. For example, as one

crewmember indicated, being able to view the location of the downwind to base turn makes it possible to

efficiently schedule approach briefings and checklist

completion without guessing how much time is available. For surface operations, flight crews reported that

having traffic information was beneficial, but such

information would be of greater value with the addition of an electronic surface map. Although flight

crews were aware of a workload cost imposed by the

addition of CDTI platforms, most indicated that this

cost might be mitigated or eliminated with more

experience, better procedures design, and training.

Training

Flight crews using the Build 2/4 platform provided

most of the comments in the training category. Several comments indicated that the training for the

OpEval-2 applications was inadequate and there were

many suggestions for improvement. The most frequent comment was a desire for hands-on practice

with the platform, preferably in the workload context

of a full mission simulation or line-oriented, flighttraining scenario. Short of this, a desktop simulation,

dedicated CDTI training aids, or, as suggested by

more than one pilot using the MX-20 platform, a CDROM demonstration of acceptable procedures should

be provided. Additional training would be very beneficial to crew performance and it could significantly

reduce workload. Evidence for this statement comes

from the frequent observation by flight crews and

observers that there was a noticeable learning curve

from one flight period to the next. These improvements appeared to be due to flight crews’ exposure to

CDTI platforms and OpEval-2 applications. The

exact form of an appropriate training program must

be determined by operators and approved by the FAA.

ATC Interaction

Regardless of the CDTI platform, the majority of

flight crew responses in this category pointed to a

need for exact specification of procedures for flight

crews and controllers governing the use of CDTI.

Advanced Prototype flight crews also noted that it

would be very important for controllers to be familiar

with CDTI capabilities and limitations. Spacing techniques by flight crews using CDTI may result in

observed aircraft behavior that differs substantially

from controller spacing techniques. Flight crews also

noted that some of the OpEval-2 restrictions on

CDTI-based maneuvering limited their ability to

effectively use CDTI to accomplish spacing objectives. In particular, the prohibition on lateral maneuvers, combined with short leg lengths, made it difficult

to achieve spacing goals using speed control alone.

Flight crews using the Advanced Prototype platform

indicated that it would be very important to have

procedures in place that specify when and where flight

crews would assume responsibility for tasks associated

with the Final Approach Spacing application. Their

comments included a broad range of suggestions,

including improved phraseology, explicit spacing assignments, and the possible creation of ‘electronic

flight rules’ to govern the integration of final approach spacing tools with existing ATC procedures.
21

Several flight crews suggested that the general knowledge of the traffic situation due to CDTI reduced the

level of ATC communication. Others reported more

accurate visual search for called traffic as a benefit.

Applications

Comments on applications came from the flight

crews using the CDTI Build 2/4 and Basic Prototype

platforms. In general, they indicated that Departure

Spacing was easily accomplished and that Initial Spacing using CDTI was accurate and provided for earlier

detection of spacing error than would be possible with

spacing provided by ATC. Flight crew comments also

indicated that the absence of an electronic surface

map made it difficult, especially with existing display

resolutions, to determine directly from the display

whether or not a runway was occupied. On the other

hand, observer comments indicated that flight crews

did become aware of possible incursion traffic by

referencing CDTI, which resulted in increased visual

search out the window to locate traffic.

Comments by flight crews using the CDTI Build

2/4 platform for tasks associated with the Final Approach Spacing application suggest that this task was

difficult under OpEval-2 conditions. The restriction

on lateral maneuvering reduced the ability of flight

crews to use CDTI for establishing position to complete the spacing task. When forced to use speed

alone, they expressed concern about the possible negative impact on a stabilized final approach. Finally,

several comments related to the need for an electronic

surface map to support the Airport Surface Situation

Awareness application. Flight crews commented that

although the display of other traffic is useful, without

an electronic surface map there is little support for

runway incursion prevention.

Post-Flight Flight Crew Questionnaires

Two post-flight questionnaires, one for the MX-20

platform and one for the three other CDTI platforms,

were administered after flight crews completed participation in their final OpEval-2 flight period. The questionnaires included statements about OpEval-2

applications and CDTI platform usability. These statements were representative of the following topics:

• Training - previous experience with and operation of

platform.

• OpEval-2 Applications - use of CDTI platform for

ASSA, Departure Spacing, Traffic Awareness, Initial

Approach Spacing, Final Approach Spacing, and

FAROA.

• Platform Functions - ease of use of functions during

operation (e.g., Map Range, Target Select, etc).

• Platform Color and Symbology - assessed meaningfulness of display colors and symbols during operation of the platform.

• Platform Features - assessed utility of platform functions, as well as flight crew perceptions of workload,

heads-down time and effect on decision making.

• Location and Readability - assessed location and

integration of the platform into the cockpit display

suite, and assessed issues associated with legibility.

Analysis of data from the questionnaires included

frequency counts, descriptive statistics and analysis of

variance on summary data, which was derived by

combining groups of ratings for OpEval-2 applications. In addition, factor analyses were used to determine if usability ratings for usability statements could

be grouped according to some underlying factor.

Summary data was computed for each factor and

analysis of variance was performed to determine if

OpEval-2 platform types varied along these factors.

Responses to open-ended questions were analyzed

to identify other issues that flight crewmembers

thought were important. Open-ended responses from

the two questionnaires were aggregated with responses

from the structured interview to summarize the operational and human factors issues that flight crews identified for each platform type and OpEval-2 application.

Flight Crew Training

Thirty-four percent of the flight crews participating in Op-Eval-2 had some previous experience flying

a CDTI-type platform. Approximately 82% of these

flight crews stated that they flew CDTI on a test

flight. This experience may have been gained during

OpEval-1, in which some of the OpEval-2 flight

crews participated. The mean number of hours flown

using CDTI-type platform ranged from 1.0 hr to 275.0

hrs, with a mean of 19.1 hr and a median of 5.0 hr.

Table 5 shows the different methods of CDTI-type

training to which flight crews were exposed. The ILAB training consisted of simulation training at MITRE CAASD in preparation for OpEval-2. OpEval-2

training included the briefings provided flight crews

immediately before OpEval-2 began. Formal training

included classroom and flight instruction (simulation). Informal training included reading the platform-operating manual and other training included

all types of training not covered by the training types

previously listed. Table 5 reveals that flight crews who

participated in OpEval-2 received most of their training from OpEval-2 briefings and from informal methods, such as reading the platform operating manual.
22

OpEval-2 Applications

Flight crews rated their level of agreement to each

questionnaire statement by using Likert-type scales

(1=Strongly Disagree, 2=Somewhat Disagree, 3=Neither Agree Nor Disagree, 4=Somewhat Agree,

5=Strongly Agree). Pilot ratings were converted to a

favorable - unfavorable scale to simplify interpretation. Ratings less than 2.75 were considered unfavorable and ratings greater than 3.25 were considered

favorable.

Figure 14 presents the average pilot rating for each

OpEval-2 application. The ratings are classified further by the CDTI platform type. The average rating

for an application was derived by summing all ratings

for that application and dividing this sum by the total

number of ratings for that application. This method

was used after analysis of ratings for individual questionnaire statements showed minimal variation in

flight crew responses. A one-way analysis of variance

(ANOVA) was performed for each application, using

CDTI platform type as the independent variable and

average rating for each OpEval-2 application as the

dependent variable.

Table 5. Summary of CDTI training listed by OpEval-2 flight crews. 

 Rating of Training by Flight Crews 

Training

Percent of 

Flight Crews 

Participating

Not 

Helpful 

Somewhat 

Helpful 

Very 

Helpful 

Average/Median 

Number of 

Training Hours 

Range of 

Training 

Hours 

I-LAB 

Simulations 36% 36% 64% 0% ----- ----- 

OpEval-2 

Training 50% 6% 19% 75% 3.6 / 3.0 1.0 – 7.0 

Formal Training 28% 11% 22% 67% 2.1 / 1.5 1.0 – 8.0 

Informal Training 78% 3% 64% 33% 3.8 / 2.0 1.0 – 20.0 

Other 19% 17% 17% 67% 0.2 / 0.10 0.05 – 0.5 

Figure 14. Average pilot rating for each OpEval-2 application by CDTI platform type. 

FAROA

Final Approach Spacing

Initial Approach Spacing

Traffic Awareness

Departure Spacing

ASSA

MX20 (No spacing tasks)

Advanced Prototype

Basic Prototype

CDTI Build 2/4

Strongly

Agree

Somewhat 

Disagree

Neither Agree 

Nor Disagree

Somewhat

Agree

Strongly

Disagree
23

Table 6 includes the results of post-hoc tests for

each of the OpEval-2 application average ratings. The

∩ indicates that one CDTI platform type was rated

significantly higher than the CDTI platform type(s)

with ∪ in the same row. For example, the ∩ and ∪ in

the row for the ASSA application indicate that the

CDTI Build 2/4 platform was rated significantly

higher than the Basic Prototype platform. The MX20 platform was used only for the ASSA and FAROA

applications and to evaluate traffic awareness. It was

not used during any of the spacing applications.

ASSA

The ASSA application portion of the questionnaire

included statements that elicited flight crew responses

about how well the CDTI Build 2/4, Basic Prototype

and Advanced Prototype platform functions and features supported surface awareness, taxiing, and acquisition of ground traffic. The results of the ANOVA

indicated that the highest ratings were associated with

the CDTI Build 2/4 and MX-20 platforms, as shown

in Figure 14. The flight crews agreed somewhat that

these two platform types enhanced ASSA. Table 6

reveals that both the CDTI Build 2/4 and MX-20

platform were rated significantly higher than the

Basic Prototype platform. All platform types received

favorable ratings except the Basic Prototype platform.

Examination of the individual item responses, however, indicated that most flight crews found clutter to

be an issue for the ASSA application.

Departure Spacing (Not Applicable to MX-20)

The Departure Spacing application portion of the

questionnaire included statements that elicited flight

crews’ responses about how well the CDTI Build 2/4,

Basic Prototype and Advanced Prototype platform

functions and features supported climb-out, traffic

awareness, and range identification tasks during departure spacing. Figure 14 shows that the CDTI Build

2/4 and Advanced Prototype platform received favorable ratings. Although the Basic Prototype was not

rated unfavorable, it did receive the lowest overall

Table 6. Tukey HSD post-hoc tests for flight crew questionnaire summary ratings by platform type and 

OpEval-2 application or Usability Factor.1

OpEval-2 Application/Usability Factor 

CDTI 

Build 2/42

Basic 

Prototype 

Advanced 

Prototype MX20 

ASSA  n.s. 

Departure Spacing  n.s. ----- 

Traffic Awareness  n.s. 

Initial Approach Spacing  n.s. ----- 

Final Approach Spacing n.s. n.s. n.s. ----- 

FAROA  n.s. 

Display Functions  n.s. ----- 

 Color and Symbology n.s. 

Decision Making and Task Completion  n.s. 

Display Information  n.s. ----- 

Head-Down Time and Workload n.s. n.s. 

Aircraft Speed/Vector Information  n.s. ----- 



Readability and Display Location 

 

Button Labeling, Layout and Size n.s. 

Symbol Size  n.s. ----- 

Button Cycling (e.g., Target Range/Selection)  ----- 

1 All post-hoc comparison tests were conducted using a significance criterion of 0.05. “” indicates that a given platform type was rated 

significantly higher than the platform type(s) with “” in the same row. Within rows, platform types with arrows in the same direction 

(e.g., CDTI Build 2/4 and MX-20 platforms for ASSA) are not significantly different. “n.s.” indicates that a given platform type does not 

differ from the other platform types in the same row. “-----” is used to indicate that the MX-20 was not used for an OpEval-2 application 

or that the MX-20 was not rated on a given factor. Cells with “gray shading” indicate that more than one row was needed to explain the 

relationship between the ratings for platform types. For example, the comparisons for the “Color and Symbology” factor could not be 

represented using a single row. As shown in the first row for this factor, CDTI Build 2/4 platform ratings were significantly higher than 

those of the Basic and Advanced Prototypes. However, CDTI Build 2/4 platform ratings did not differ significantly from those of the 

MX-20, which, as shown in the second row for this factor, were significantly higher than ratings for the Basic Prototype. 

2 The CDTI Build 2/4 platform consistently was rated higher than the Advanced Prototype platform; however, statistical tests that included 

the latter platform had very low power. 
24

rating for the Departure Spacing application. Also,

Table 6 reveals that the ratings for the Basic Prototype

platform were significantly lower than those for CDTI

Build 2/4 platform.

Visual Acquisition and Traffic Awareness

The Visual Acquisition and Traffic Awareness application portion of the questionnaire included statements that elicited flight crews’ responses about how

well the CDTI Build 2/4, Basic Prototype, Advanced

Prototype, and MX-20 platform functions and features supported locating and identifying proximate

traffic. Figure 14 shows that the CDTI Build 2/4 and

MX-20 platform received very favorable ratings,

whereas the Basic Prototype and Advanced Prototype

platform did not. Table 6 indicates that the Traffic

Awareness ratings associated with the Basic and Advanced Prototype platform types were significantly lower

than the ratings for the CDTI Build 2/4 platform.

Initial Approach Spacing (Not Applicable to MX-20)

The Initial Approach Spacing application portion

of the questionnaire included statements that elicited

flight crews’ responses about how well the CDTI

Build 2/4, Basic Prototype, and Advanced Prototype

platform functions and features supported target selection, station keeping, and estimating distance and

speed. Although Figure 14 shows that the CDTI

Build 2/4 and the Basic Prototype platform received

favorable ratings, the CDTI Build 2/4 platform was

rated the highest. Table 6 indicates that the rating for

the CDTI Build 2/4 platform was significantly higher

than the rating associated with the Basic Prototype

platform. The ratings for the Basic Prototype and

Advanced Prototype platform were equivalent.

Final Approach Spacing (Not Applicable to MX-20)

The primary objective of the Final Approach Spacing application was to minimize the variability in

spacing between aircraft on approach. The Final Approach Spacing application portion of the questionnaire included statements that elicited flight crews’

responses about how well the CDTI Build 2/4, Basic

Prototype, and Advanced Prototype platform functions and features supported target selection, station

keeping, and estimating distance and speed. Figure 14

shows that the ratings for this application look very

similar to the ratings for the Initial Approach Spacing

application. The CDTI Build 2/4 and Basic Prototype platform received favorable ratings, whereas the

Advanced Prototype platform did not. Table 6

indicates that, although the ratings for the CDTI

Build 2/4 platform were higher than those for the

Basic and Advanced Prototype platform, these differences were not significant.

FAROA

The FAROA application portion of the questionnaire included statements that elicited flight crews’

responses about how well the CDTI Build 2/4, Basic

Prototype, Advanced Prototype and MX-20 platform

functions and features supported awareness, target

detection in the runway environment, transition procedures, and estimation of runway threshold. The

results of the ANOVA indicated that the highest

ratings were associated with the CDTI Build 2/4 and

MX-20 platforms. This finding is shown in Figure 14.

The flight crews somewhat agreed that these two

platform types enhanced final approach runway occupancy awareness. Table 6 reveals that ratings for the

CDTI Build 2/4 and MX-20 platforms were significantly higher than ratings for the Basic Prototype

platform. Ratings of the Basic Prototype platform

were unfavorable. In fact, 80% of the flight crews gave

an unfavorable rating of this platform type.

CDTI Platform Usability

Exploratory principal components analysis (using

varimax rotation with Kaiser normalization) was used

to analyze the pilot ratings of statements in the questionnaire. Principal components analysis is a statistical technique that is applied to a set of variables, such

as questionnaire statements, to discern which variables in the set form coherent subsets, each of which

are relatively independent of one another. Variables

within subsets are correlated; however, subsets are

largely independent of other subsets. For example, a

group of statements within a section of the flight crew

questionnaire (e.g., Location and Readability section)

may be highly correlated and independent of other

groups of statements within the same section. The

correlation among statements within a given subset is

interpreted as an underlying component or factor that

can be used to describe the subset of statements.

Principal components analysis was used in the present

context to reduce a large number of questionnaire

statements to a smaller number of usability factors on

which OpEval-2 platform types could be compared. In

addition, the definition of factors yields information

that can be used to provide human factors guidance

during the certification process (e.g., factors become

checklist items in an HF supplement to a TSO).
25

Four principal components analyses were performed, one for each of the ergonomics sections in the

questionnaire. The results of these analyses are summarized in the following list, which identifies 10

factors:

1: Display functions

2: Consistent and acceptable color and symbology

3: Enhanced decision making and task completion

4: Understanding of display information

5: Head-down time and workload

6: Display of aircraft speed information

7: Readability and display location

8: Button labeling, layout and size

9: Symbol size

10: Button cycling

Summary data were computed for each factor, and

analysis of variance was performed to determine if

OpEval-2 platform types varied along these factors.

Figure 15 presents the average pilot rating for each

usability factor. The ratings are further classified by

platform type. The average rating for a given factor

was derived by summing all ratings for that factor and

dividing by the total number of ratings for that factor.

Factor 1: Display Functions (Not Applicable to MX20). Flight crews were asked to rate the ease of use of

particular display functions (e.g., display mode, target select, etc.) using a five-point, Likert-type scale

(1= Difficult to use, 5=Easy to use) for questionnaire

items 113 - 120. Figure 15 shows that flight crews

rated the CDTI Build 2/4 platform easiest to use

relative to the Basic and Advanced Prototype platform. Table 6 reveals that the CDTI Build 2/4 platform was rated significantly easier to use than the

Basic Prototype platform.

Factor 2: Color and Symbology. The Color and

Symbology portion of the questionnaire, which included items 122 - 128, elicited flight crews’ responses about the appropriateness of the color and

symbology used in the displays of the CDTI Build 2/

4, Basic Prototype, Advanced Prototype and MX-20

platform. These items ranged from symbology scaling

issues to consistency of color relative to other flightdeck

displays. Figure 15 shows that the ratings for this

usability factor were favorable for the CDTI Build 2/

4 and MX-20 platforms and unfavorable for the Basic

and Advanced Prototype platforms.

Table 6 indicates that ratings for the CDTI Build

2/4 and MX-20 platforms were significantly higher

than the ratings for the Basic Prototype platform. In

addition, the ratings for the CDTI Build 2/4 platform

were significantly higher than the ratings for the

Advanced Prototype platform. Written observations

suggested that, overall, some flight crews reported difficulty in color discrimination between cyan and green.

Factors 3-6: Display Functions. Factor analyses of

pilot ratings defined four underlying factors in this

section of the questionnaire. Factor 3 included questionnaire items 142, 143 and 146, and this factor

identified decision making and task completion issues. Figure 15 shows that ratings for the CDTI Build

2/4, Advanced Prototype and MX-20 platforms all

were favorable. The rating for the Basic Prototype

platform was unfavorable. The ANOVA results in

Table 6 revealed that, for factor 3, the CDTI Build 2/

4 and MX-20 platforms were rated significantly more

favorable than the Basic Prototype platform.

Factor 4 included questionnaire items 132 -135,

and was defined as types of display information (e.g.,

data tags, traffic altitudes, etc). This factor was not

applicable to the MX-20 platform. Table 6 shows that

the ratings for the CDTI Build 2/4, Advanced Prototype, and Basic Prototype platforms were favorable.

Table 6 indicates that, for factor 4, the CDTI Build 2/

4 platform was rated significantly more favorable than

the Basic Prototype platform.

Factor 5 included questionnaire items 136, 144

and 145, and this factor revealed attentional issues

(e.g., head-down time and workload). Figure 15 shows

that only the MX-20 platform was rated favorably for

this factor. Ratings for the CDTI Build 2/4, Basic

Prototype, and Advanced Prototype platforms were

unfavorable. Table 6 shows that, for Factor 5, the

MX-20 platform was rated significantly more favorable than the Basic Prototype platform. These findings may indicate that, relative to the MX-20 platform,

the CDTI Build 2/4, Basic Prototype and Advanced

Prototype platforms were more novel to flight crews.

Regardless of the explanation, issues related to attention should be explored and attempts should be made

to mitigate the possibility of flight crews attending

too much to OpEval-2 platform types.

Factor 6 included questionnaire items 131, 137

and 139. These items focused on display of aircraft

speed and vector information (e.g., ground speed

tag). This analysis was not applicable to the MX-20

platform. Figure 15 shows that ratings for the CDTI

Build 2/4 platform were favorable, whereas ratings for

the Basic Prototype and Advanced Prototype platforms

were unfavorable. Table 6 indicates that, for factor 6,

the CDTI Build 2/4 platform was rated significantly

more favorable than the Basic Prototype platform.
26

Factors 7-10: Location and Readability. Factor analyses of pilot ratings also defined four underlying factors in this section of the questionnaire. Factor 7

consisted of questionnaire items 148, 150, 157 - 159,

and 163 - 166. This factor seemed to reveal issues

related to readability and display location. Figure 15

shows that ratings for the CDTI Build 2/4 and MX20 platforms were favorable, whereas the ratings for

the Basic Prototype and Advanced Prototype platforms

were unfavorable. The ANOVA results in Table 6 revealed that, for factor 7, ratings for the MX-20 platform

were significantly more favorable than the other three

platforms. In addition, the ratings for the CDTI Build

2/4 platform were significantly more favorable than the

Basic and Advanced Prototype platforms.

Factor 8 was defined by questionnaire items 153 -

156 and 167, all of which identified control button

features (e.g., button labeling, button layout, and

button size). Figure 15 again shows that the ratings for

the CDTI Build 2/4 and MX-20 platforms were

favorable, whereas the ratings for the Basic Prototype

and Advanced Prototype platforms were unfavorable.

The ANOVA results in Table 6 revealed that, for

factor 8, ratings for the MX-20 platform were significantly more favorable than the ratings for the Basic

and Advanced Prototype platforms.

Factor 9 consisted of questionnaire items 149 and

160 -162. These items identified issues related to

display symbol size. The analysis was not applicable to

the MX-20 platform. Figure 15 shows that the CDTI

Build 2/4 platform was rated as favorable. Table 6

shows that, for factor 9, the CDTI Build 2/4 platform

was rated significantly more favorable than the Basic

Prototype platform.

Factor 10 included questionnaire items 151 and

152. This factor, although difficult to interpret, appeared to characterize issues associated with button

cycling during target and range selection. The analysis

was not applicable to the MX-20 platform. Figure 15

shows that ratings for the CDTI Build 2/4 platform

were favorable, whereas ratings for the Advanced

Prototype platform were unfavorable. Table 6 indicates that, for factor 10, the CDTI Build 2/4 platform

was rated significantly more favorable than both the

Basic and Advanced Prototype platforms.

Figure 15. Average pilot rating for each usability factor by platform type.

1 0 . Acceptab le butto n

cycling (e.g ., targ et and

rang e s electio n)

9 . Ac c e p t ab le s ymb o l s iz e

8 . Accep table butto n

labeling , layo ut and s ize

7 . Ac c e p t ab le re ad ab ilit y

and d is play lo catio n

6 . Dis p lay o f aircraft s p e ed

inf o rmat io n w as us e f ul

5 . No increas e in head -

d o w n time and w o rklo ad

4 . Und ers to o d d is p lay

inf o rmat io n

3 . Enhanced decis io n

making and t as k c o mp le t io n

2 . Co ns is tent and

accep tab le co lo r and

s ymb o lo g y

1 . Eas y-to -us e dis play

functio ns

MX20

Ad vanced Pro to typ e

B as ic Pro to t yp e

C D TI B ld 2 / 4

Strongly 

Agree 

Somewhat 

Disagree 

Neither Agree 

Nor Disagree

Somewhat 

Agree 

Strongly 

Disagree 
27

CONCLUSIONS

The OpEval-2 flightdeck observer data supplements a corpus of evidence that contributes to the

identification and resolution of human factors issues

associated with the use of CDTI by flight crews. The

following conclusions and recommendations, which

are organized by OpEval-2 applications and CDTI

certification issues, are drawn from analysis of the

flightdeck observer data.

OpEval-2 Applications

ASSA

While navigating the airport surface along complex

routes, flight crews who used electronic surface maps

were able to easily perform taxi operations and identify taxiway locations relative to the crews who used

paper surface maps. Nearly 67% of the flight crews

with electronic surface maps used them for awareness

of ownship position on the airport surface, and they

were able to identify their location on the airport

surface whereas crews with a paper surface map were

not. All but one of the flight crews with an electronic

surface map used the CDTI to support surface taxi

operations at least some of the time. Also, flight crews

unanimously agreed that awareness of airport surface

operations was enhanced when the CDTI platform

included an electronic surface moving map. Taken

together, these findings indicate that a CDTI showing the position of own aircraft relative to an airport

map appears to improve geographical and situation

awareness.

Flightdeck observer data revealed that flight crews

with electronic surface moving maps committed fewer

errors than did flight crews with paper surface maps.

Furthermore, when a taxi error was made, flight crews

with electronic surface maps also were able to rejoin

their assigned structured routes more readily than

those with paper surface maps. Although statistical

analysis of errors was not possible because there were

too few, this evidence suggests that flight crew awareness can be enhanced by a CDTI with an electronic

surface map. This evidence is compelling given that

flight crews with paper surface maps were much more

familiar with the airport used in this study than those

with electronic surface maps.

Flightdeck observers reported that a higher percentage of crews with an electronic surface map used

it to enhance awareness of the airport surface relative

to those with a paper surface map. This enhanced

awareness was accompanied by an increase in the use

of the electronic surface map to assess traffic location

on the airport surface, possible conflicts during taxi

along the structured routes, locating TTF, and for

general traffic awareness. Post-flight interviews indicated that the addition of traffic information was

valuable for the ASSA application, and that this information was enhanced with the addition of an electronic surface map. When asked whether CDTI aided

in traffic awareness, flight crews who used an electronic surface map agreed more strongly that it did

relative to flight crews with a paper surface map.

In their responses to the post-flight questionnaires,

all flight crews agreed that CDTI (with and without

an electronic surface moving map) enhanced airport

surface situation awareness. Overall, flight crews using the MX-20 platform, which included the moving

map, did provide higher ratings for ASSA, but the

difference was not statistically significant.

Flightdeck observer data indicated that the CDTI

function known as Target Selection was used by

approximately the same percentage of crews with

electronic surface maps as those with paper maps.

The function was used to aid surface navigation. The

Ground Track Vector function was used by few crews

during any task, regardless of whether these crews had

an electronic surface map. The Range function was

used twice as often by crews with electronic surface

maps as crews with paper surface maps. These data

indicate that, for the ASSA application, the Target

Selection function may be valuable for CDTI with or

without an electronic surface map, the Range function may be particularly valuable for CDTI with an

electronic surface map, and the Ground Track Vector

function does not appear to be useful. Flight crew

comments from post-flight interviews indicated that

use of the CDTI for the ASSA application could be

enhanced by developing an effective means to control

clutter, displaying runway and taxiway markings,

using a track-up map orientation, providing NOTAM

data (e.g. closed runways and taxiways), and including a pan and zoom function.

Flight crews were given a high-workload, surface

navigation task during which their navigation, geographic and traffic awareness were evaluated. Flight

crews that used CDTI platforms with electronic airport surface maps and those who used paper surface

maps both agreed that it was easy to get oriented to the

CDTI. These crews also reported that CDTI aided

the tasks of locating and determining the positions of

traffic, and that it aided their understanding of ATC

communications. These findings imply that CDTI

reduced the workload of flight crews during airport

surface operations. Additionally, there was almost
28

unanimous agreement by flight crews on the need for

an electronic surface map on the CDTI to support the

ASSA application.

Departure Spacing

The Departure Spacing application demonstrated

that flight crews using CDTI and controllers using

current procedures without ADS-B could minimize

spacing variation and that it would not affect their

workload. For example, post-flight interview data

suggested that flight crews generally thought that

tasks in the Departure Spacing application were easy

to perform. This finding suggests that the Departure

Spacing application would not affect flight crew

workload. However, more research is required to

determine the impact on airport capacity and safety of

increased “runway ownership time,” which represents

the time allotted by the controller to the flight crew to

maneuver the aircraft onto the runway and complete

the departure roll after receiving a take off clearance

from the controller. During this time, the runway is

unavailable to the controller for vehicle or other

aircraft operations (e.g., runway crossing, landings,

etc.). Additionally, there is an increase in the potential

for an accident because the aircraft remains on the

active runway until appropriate spacing is achieved.

Initial and Final Approach Spacing

During the Initial Approach Spacing application,

flightdeck observers reported high rates of compliance with OpEval-2 recommended CDTI function

settings for Range Ring (79%), Vector Time (86%)

and Select Target (78%). In many cases, these settings

were made previously and simply retained by flight

crews. Although adherence to these recommendations does not necessarily validate their use for initial

approach spacing, it at least implies that the flight

crews did not prefer other settings. These findings

indicate that for initial approach spacing, flight crews

were able to use the CDTI with the range ring, vector

time, and target select functions. It should be noted

that further work is needed before initial approach

spacing can be performed by flight crews in the NAS.

Flightdeck observers also found that 85% of flight

crews used the Select Target function during the Final

Approach application. Flight crews and observers

both reported that this function was the crews’ primary source of information. Only half the crews

complied with the Vector Time setting recommendation and this finding likely was due to the incompatibility of the vector indications (i.e., time) with the

distance-based information provided for the spacing

task (i.e., distance). The recommended Range Ring

setting was observed 82% of the time during the Final

Approach Spacing application.

Analysis of flightdeck observer data indicated that

pilot workload during the Initial and Final Approach

spacing applications varied by CDTI platform type.

Cooper-Harper workload ratings for flight crews using the CDTI Build 2/4 CDTI platform revealed that

mental effort was required to attain acceptable performance. Ratings associated with the Advanced Prototype platform indicated that high mental effort was

needed to attain acceptable performance. Ratings for

the Basic Prototype platform indicated that maximum mental effort was required to avoid large or

numerous errors. Although these results indicate that

the implementation of the Basic Prototype platform

resulted in excessive workload levels, design efforts to

reduce the workload associated with all platform

types would be beneficial to flight crews.

During Final Approach Spacing, flight crews using

the CDTI Build 2/4 platform reported competition

for attention between the CDTI and the HUD, which

provided primary flight control information. They

commented in interviews that PNF workload was

higher than would be acceptable in line operations,

and that use of the CDTI caused distraction from

other tasks, such as hearing ATC communications,

monitoring airspeed and altitude, and checklist

completion. Written comments by flight crews suggested that the use of CDTI resulted in two of the five

checklist suspensions and five of the ten checklist

interruptions.

Flight crews using the Advanced Prototype platform reported higher than expected workload during

the Final Approach Spacing application. The higher

workload was attributed to the frequency and magnitude of speed adjustments required by the advanced

spacing algorithm commands. Reducing the frequency

and magnitude of these speed commands may reduce

the mental effort currently required to achieve acceptable performance. The relay of spacing information

from PNF to PF created higher workload for the PNF.

This issue could be mitigated by relocating the spacing

information within the PF’s primary field of view

where the PF would be able to access it directly. Even

with this solution, the CDTI-related responsibilities

for the PF and PNF need to be integrated into existing

flightdeck checklist procedures for the respective

phases of flight.
29

FAROA

The FAROA Air application required flight crews

to use CDTI while on final approach to determine

whether the runway was occupied. During some restricted low approaches, vehicles were positioned on

the runway. Eight flight crews with electronic surface

maps identified vehicles were occupying the runway,

whereas none of the crews with paper surface maps

were able to identify these vehicles. Although these

data appear to indicate that a CDTI with an electronic

surface map enhances the ability of flight crews to

detect runway intrusions, an appropriate comparison

would be needed to establish the efficacy of using

CDTI with a paper surface map for FAROA Air.

In their responses to the post-flight questionnaire,

the flight crews using the CDTI Build 2/4 and MX20 platforms agreed that the CDTI supported the

FAROA Air application. The MX-20 platform included an electronic surface map, whereas the CDTI

Build 2/4 platform did not. Flight crews using a

CDTI platform that included an electronic surface

map agreed that the CDTI aided in detecting traffic

crossing the runway, traffic near the runway, traffic

on the runway, and gauging when traffic exited the

runway. Flight crews using a CDTI with a paper

surface map agreed that the CDTI aided in determining when traffic was on the runway. Data from postflight interviews indicated that flight crews noted

difficulty in determining runway occupancy directly

from the electronic surface map. Anecdotal data from

flightdeck observers’ suggested that flight crews became aware of possible incursion traffic by reference

to the CDTI without an electronic surface map, after

which they increased their visual search out the window to locate the traffic.

Overall, flight crews generally agreed that CDTI

aided in taxiway, runway, final approach, and departure traffic awareness during takeoff. Additionally,

they agreed that the color change from brown (on the

surface) to green (when airborne) aided in awareness

of aircraft status. Flightdeck observers reported evidence indicating that substantially more crews with

an electronic surface map than crews without an

electronic surface map used CDTI to assess runway

occupancy as well as the approach and the departure

corridors. These findings indicate that surface map

capability appears to increase the use of CDTI for

Final Approach Runway Occupancy Awareness

(FAROA) Air and Ground.

The CDTI proved valuable for seeing properly

equipped aircraft on or near the runway during the

takeoff and approach phases of flight. However, as the

TTF approached the runway, the symbol that represented it on the display merged momentarily with

other aircraft symbols that were grouped as surface

targets. Also, the absence of an electronic airport

surface map and the inability to determine the precise

position of traffic relative to the runway environments were problematic for flight crews on final

approach. Many flight crews overcame these problems by using the Range function to manually adjusting CDTI range to one or two miles after passing the

final approach fix, which is normally a time when

crews devote full attention to landing.

Flightdeck observers also noted that the flight

crews performing the FAROA Ground and Air applications with electronic surface maps used the CDTI

to evaluate approach and departure corridors, and

runways for possible conflicts. During final approach

when flight crews were performing the FAROA Air

application, observers reported that crews with an

electronic surface map used different surface map

modes. Although many crews used the Compass Rose

mode and some used the Arc mode, most crews with

a paper surface map used the Arc mode. The Range

function was used to adjust the map range by most

crews, especially those with electronic surface maps. A

relatively small number of crews with and without

electronic surface maps used the Ground Track Vector function for runway occupancy awareness during

approach to display target speed. These data indicate

that, for the FAROA Air application, Track-up 360

mode and the Range function may be used with an

electronic surface map more than other modes, Arc

mode may be used without an electronic surface map

more than other modes, and that the Ground Track

Vector function may be used both with and without

the electronic surface map.

Visual Acquisition and Traffic Awareness

As much as possible, flight operations using CDTI

in a mixed equipage environment should include the

display of all aircraft on the CDTI. In post-flight

structured interviews, a large majority of flight crews

commented that the display of all aircraft would be

necessary to achieve the full benefit of CDTI. One

crewmember using the Advanced Prototype platform,

which displayed all OpEval-2 aircraft, commented

favorably on the utility of this feature. Some flight

crews suggested that they would be required to maintain a robust visual search for aircraft that are not

displayed in mixed equipage conditions.
30

CDTI Certification

One goal of the OpEval-2 test was to provide

information to avionics manufacturers and the FAA

Aircraft Certification Office to support certification

of CDTI. For air transport category aircraft, the

primary certification requirements are defined by the

Code of Federal Regulations 14 Part 25. Part 25

addresses two primary issues: (1) Does the platform

perform the intended function? and (2) Could the

platform lead to hazardous situations? The following

paragraphs summarize flightdeck observer data from

OpEval-2 that will be used to address the two primary

certification issues contained in Part 25.

Usability

Overall, flight crews rated the CDTI Build 2/4 platform functionality as easy to use and agreed that it

enhanced decision making and task completion. Display

information was considered understandable; symbol

size, symbology, button cycling were considered acceptable; and, use of color was considered consistent and

acceptable. Responses to the flight crew questionnaire

for the MX-20 platform also indicated that it enhanced

decision-making and task completion, the readability

and display location, and that the button labeling,

layout, and size were acceptable.

The Advanced and Basic Prototype CDTI platform

were developed specifically for OpEval-2. The Advanced

Prototype platform readability and display location were

rated unfavorably; particularly its button labeling, layout, and size. The Basic Prototype platform readability

and display location also were rated unfavorably. These

responses may be related to the display-control interface

and not the actual display interface. Flight crew questionnaire responses suggested that workload and headdown time increased most with the Advanced and Basic

Prototype platforms, although responses indicated that

they increased with the CDTI Build 2/4 platform as

well. Collectively, these findings indicate that several

usability issues remain with the Basic and Advanced

Prototype platforms, which is to be expected given that

both systems were still in the developmental stage.

CDTI in Flight Crew Field of View

An important certification issue pertains to whether

the CDTI platform should be included in the flight

crews’ primary scan pattern. Aircraft participating in

OpEval-2 had the CDTI platform mounted in the

front panel of the flightdeck. The platform appeared

to be scanned easily for all applications tested. However, none of the aircraft had the information from

CDTI integrated with the Electronic Horizontal Situation Indicator or a Head-up Display (HUD), which

are in the flight crews’ primary scan pattern. Flight

crews commented on this fact and they noted that

traffic integration with the NAV display would provide the most effective tactical information for situation awareness. Furthermore, flight crews commented

that the CDTI platform was out of the primary scan

pattern during the Final Approach application. Apparently, the HUD, which was providing primary

flight control information, and the centrally located

CDTI competed for flight crews’ attention. This

phenomenon occurred despite a recommended technique that instructed only the PNF to attend to the

CDTI platform. Finally, some flight crews noted that

it was difficult to observe CDTI information from

other locations on the flightdeck.

Display Presentation

Another important certification question pertains

to CDTI platform display size and other display

attributes that supported each OpEval-2 application.

Data relevant to these issues were collected during

OpEval-2 using post-flight, structured interviews and

flight crew questionnaires. Overall, flight crews listed

the following changes, which they thought would

enhance display attributes such that the CDTI platform could be used efficiently for the applications

tested during OpEval-2.

• A track-up, surface moving map and taxiway labels

that are always in view and properly oriented.

• Better color differentiation- Selected target vs. background; also, written observations suggested that

some flight crews reported difficulty in discriminating between cyan and green.

• Display feature priority (e.g., modify priorities for

overlaying targets).

• Reduce size of ownship

• Include ownship velocity vector

• Develop a simpler means for de-cluttering display.

• Include target selection capability and display associated information in all traffic modes.

Crew Workload

Another important certification issue, crew

workload, was assessed using the Modified CooperHarper workload scale, and by including several questions on the flight crew questionnaire related to

workload (see items 67, 68, 70, 89, 90, 94, and 145 in

Appendix B). The Cooper Harper ratings are valuable

in that they partially address the flight crew workload
31

certification issue. Comparisons of Cooper-Harper

workload ratings and the overall average of the seven

flight crew questionnaire items were performed to

determine the extent of workload for two- and threemember crews. These comparisons were dependent

upon and included ratings from all CDTI platform

types. The Cooper-Harper ratings for CDTI platforms indicated that three-member crews (ave = 2.1)

gave significantly lower workload ratings than twomember crews (ave = 6.5). A rating of 2.1 corresponds

with the following statement on the Modified Cooper-Harper Workload Scale: “Mental effort is low and

desired CDTI performance is attainable.” A rating of

6.5 corresponds with “Maximum mental effort is

required to attain acceptable CDTI performance.”

Interpretation of the workload ratings is accompanied by several caveats. First, although flight crews

explicitly mentioned workload in the context of the

Final Approach Spacing application, flightdeck observers suggested that the PF appeared to give an

overall rating for all of the spacing tasks. Hence,

exactly what the PF rated is not entirely clear. Second,

the workload ratings given by flight crews in two-and

three-person crews are based on different tasks and

different platforms. Furthermore, the flight crews in

these crews had differing skill levels. Finally, it is

important to remember that flight crews had minimal

training on the CDTI platform (median = 5 hrs) and

that operating environment and procedures associated with OpEval-2 were not the same as those experienced daily by line flight crews in normal revenue

operations. Most crewmembers commented that the

workload associated with each CDTI platform would

be manageable with more experience, better procedures design, and training.

CDTI Clutter

The analysis of OpEval-2 applications provided an

opportunity to evaluate the CDTI platform display

for clutter. The flight crew questionnaire contained

four items about clutter (see items 30, 31, 38, 50 in

Appendix B). Although flight crews rated these four

items favorably, suggesting that display clutter was

not an issue for the CDTI platform, some of the crews

did comment about display clutter. For example, they

took issue with the momentary merging of the TTF

target with a grouping of surface targets as TTF

approached the runway. This phenomenon complicated the FAROA Air task for the aircraft following,

and made the task of determining runway occupancy

more difficult. Display clutter issues appeared to be

dependent upon the type of OpEval-2 application; in

particular, FAROA and ASSA. Generally, flight crews

wanted a more effective way to configure the CDTI

platform such that the information necessary to support the task was displayed and that which was unnecessary was eliminated.

Another important issue associated with the ASSA

application is the evaluation of surface clutter on the

traffic display. The ASSA application portion of the

flight crew questionnaire included two statements

(see items 30 and 31 in Appendix B) about surface

clutter and its effect on flight crews’ use of the CDTI

platforms for airport surface situation awareness. Although a majority found clutter to be an issue, many

did not, and most indicated that the CDTI de-clutter

feature was effective for surface applications. However, post-flight structured interviews revealed that

flight crews were affected by surface clutter during the

FAROA Air task as previously described. This clutterrelated phenomenon should be mitigated.

Display Symbology

Display Symbology, which was identified as a usability factor, focused on color coding of traffic symbols, consistency of color and symbology usage relative

to other flightdeck displays, use of color to aid decision making, symbology used to identify ADS-Bequipped traffic, and the appropriateness of symbol

sizes, including own-ship. Overall, the ratings for the

Display Symbology factor were favorable for the CDTI

platforms. Although flight crews rated the CDTI

symbology favorably, the symbology for depicting

obstructions on the airport map should be evaluated.

No symbology for obstructions was used in OpEval2, and standard symbology is needed for representing

database obstacles.

Airport Map Attributes

Airport map database attributes must be evaluated

to validate the accuracy, resolution, and integrity of

the database. Characteristics of independently designed and developed databases (i.e., National Oceanic and Atmospheric Agency, Jeppesen, and others)

should be compared, and chart designs and their

functionality should be evaluated. Although the map

survey group at NOAA indicated that they completed

a limited validation prior to its use at OpEval-2, the

electronic airport map database was not validated

completely. Data collected during OpEval-2 ASSA

and FAROA applications were limited to measures

associated with the accuracy of aircraft position information within the aircraft map database.

Airport map database critical feature attributes

must be specified, along with appropriate levels of

accuracy, resolution, and integrity. Attributes that
32

may be supported with lesser degrees of accuracy also

must be determined. The RTCA Special Committee193 (SC-193; 2001) is developing standards for airport databases and has suggested three levels of database

accuracy: coarse, medium, and fine (see pp. 37 of SC193 standards for a description of each level). Based

on taxi task requirements for navigation and geographical orientation during OpEval-2, the fine resolution database was selected to support ASSA and

FAROA tasks. Although no specific data was collected to determine whether the “fine” level of database accuracy was required, crew responses and

performance during ASSA and FAROA applications

provides some data pertaining to this issue. For example, during surface, final approach, landing, and

departure operations, flight crews reported that the

CDTI platform generally aided both airport surface

situation and traffic awareness. Additionally, crews

with an electronic surface map reported that the

system accurately depicted their position, aided in

maintaining geographical orientation during taxi,

aided in understanding ATC taxi clearances, and

aided in determining the position of traffic on or near

runways and taxiways.

The level of accuracy and resolution represented by

critical attributes in the airport map database used in

OpEval-2 could be examined and evaluated for “completeness” by RTCA SC-193. The surface vehicle

used to test the airport map database traversed many

of the Standiford International Airport taxiways prior

to OpEval-2 and personnel who performed this testing did not report any “incomplete” features. Furthermore, flight crews who participated in OpEval-2

indicated that the display of critical airport map

features on the CDTI platform corresponded with

what they observed on the airport surface. These

findings are tempered because both flight crews and

ground survey personnel were unfamiliar with criteria

for determining which features were critical to surface

navigation and airport surface situation awareness.

Research on airport surface operations has not provided sufficient evidence to support identification of

features critical to surface navigation and airport

surface situation awareness. In an attempt to gather

more information about critical features that may

have influenced flight crew performance during

OpEval-2 ASSA applications, day and night vehicle

surveys of four basic taxi routes were completed

immediately after OpEval-2. Initial results of these

surveys led to the identification of some locations on

the airport where airport signage and taxiway lines

were either missing or confusing. Analyses of the

OpEval-2 data from the ASSA application revealed

that taxi errors were associated with airport surface

locations where information was missing. Finally,

additional research is needed to determine which

airport and database features are critical to surface

navigation and airport surface situation awareness,

and to develop a hierarchy for including these features

in electronic airport map databases.

REFERENCES

Operational Evaluation Coordination Group (2000).

CAA/FAA ADS-B/Safeflight 21 Phase 1 - Operational Evaluation 1 Final Report.

RTCA SC-186 (Sep. 13, 2000). Application Descriptions for Initial Cockpit Display of Traffic Information (CDTI) Applications. Document No.

RTCA/DO-259.

RTCA SC-186. (Nov. 12, 1999). Minimum Operational Performance Standards for Cockpit Display of Traffic Information Vers. 2.5.

Test and Evaluation Master Plan (TEMP) Ver. 2.0,

Phase II – Operational Evaluation, Louisville

(Standiford) International Airport (SDF), Louisville, KY, Fall, 2000, published October 2000.

User Requirements for Aerodrome Mapping Information (Working Draft 9.0), March 4, 2001,

produced by RTCA SC-193 and EUROCAE

WG-44.
A1

APPENDIX A

ACRONYMS

ADS-B Automatic Dependent Surveillance – Broadcast

ANOVA Analysis of Variance

ARTCC Air Route Traffic Control Center

ASSA Airport Surface Situation Awareness

ATC Air Traffic Control

CAA Cargo Airline Association

CAASD Center for Advanced Aviation System Development

CDTI Cockpit Display of Traffic Information

FAA Federal Aviation Administration

FAF Final Approach Fix

FAROA Final Approach and Runway Occupancy Awareness

GPS Global Positioning System

HUD Head-up Display

I-Lab Integration and Interaction Laboratory

LDPU Link and Display Processor Unit

NAS National Airspace System

NM Nautical Miles

NOTAM Notice to Airmen

OCG Operational Evaluation Coordination Group

OpEval Operational Evaluation

PF Pilot Flying

PNF Pilot Not Flying

SDF Louisville International Airport

SF21 SafeFlight 21

TCAS Traffic Alert and Collision Avoidance System

TEMP Test and Evaluation Master Plan

TIS Traffic Information Service

TIS-B Traffic Information Service-Broadcast

TRACON Terminal Radar Control

TSO Technical Standard Order

TTF Traffic to Follow

UAT Universal Access Transceiver

UPS AT UPS Aviation Technologies

B1

APPENDIX B: FLIGHTDECK OBSERVER DATA-COLLECTION TOOLS 

CDTI BUILD 2/4 AND PROTOTYPE PLATFORMS FLIGHT CREW QUESTIONNAIRE

See Adobe Acrobat file (CDTI Questionnaire.pdf) 

MX-20 PLATFORM FLIGHT CREW QUESTIONNAIRE 

The purpose of this questionnaire is to evaluate flight crew human factors related to the use of the MX-20 for the 

applications evaluated in the second Safe Flight 21 Operational Evaluation (OpEval-2). Results of this questionnaire 

and data collected during the OpEval-2 may be published; however, your identity will be kept confidential in 

accordance with 45 CFR Part 46 of 1981, and 49 CFR Part 11 of 19915. Neither your employer nor any regulatory 

agency shall have access to your identity, as it appears in any of the data collected from this questionnaire or other 

sources for the purposes of this study. 

Please use the following point of contact if you have questions about the data collection procedures or have related 

concerns. 

B. Oscar Olmos 

The MITRE Corporation 

Center for Advanced Aviation System Development 

1820 Dolley Madison Blvd 

McLean, VA 22102-3481 

(703) 883-5746 

Date: October , 2000 

 Example 2 5 

 

Flight Event: 

 Example 3 

 
B2

CDTI BUILD 2/4 AND PROTOTYPE PLATFORMS FLIGHT CREW QUESTIONNAIRE

Pilot Name: ___________________________________ 

 

Aircraft Tail Number: ___________________________ 

Crew Position (please indicate your response by placing a "X" on the appropriate line): 

 _____ Captain _____ First Officer 

 

Flight Crew Information 

1. FAA Certificates/Ratings (please indicate your response by placing a "X" on ALL applicable lines): 

 _____ Private 

 _____ Commercial 

 _____ ATP 

 _____ Instrument 

 _____ CFI 

 _____ CFII 

_____ Other (please write in your response →): _________________________ 

 

2. Aircraft Type Ratings (please write in your response): 

 

 

 

Estimate the number of hours flown. 

3. Total Hours 

Flown: 

 

 

4. Total VFR: 

 

5. Total IFR: 

 

6. Last 12 months: 

 

7. Last 90 days: 

Operational Experience (please estimate the percentage of time that you fly under each FAR part): 

Items 8-11 should total 100%. 

 

8. Part 

121: 

 %

 

9. Part 

135: 

 %

 

10. Part 91: % 

11. Other: %
B3

Previous MX-20/GX-60 Experience 

12. Have you flown MX-20/GX-60 equipped aircraft before this date? 

 _____ No 

 _____ Yes If so, please answer the following by placing a "X" on ALL applicable lines. 

 _____ as a test flight 

 _____ as part of revenue service 

 _____ other (please write in your response →): _____________________________ 

 

13. Please indicate the total number of hours flown using the MX-20/GX60.  

 Example 2 5 0 

Please indicate any MX-20 training you have had by putting an “X” in front of the type of training. 

_____ OpEval-2 training 

A) Was this training helpful?_____ Not helpful_____ Somewhat helpful_____ Very helpful_____ 

B) Please indicate the number of training hours. _______________ 

_____ Formal training (e.g., course) 

A) Was this training helpful?_____ Not helpful_____ Somewhat helpful_____ Very helpful_____ 

B) Please indicate the number of training hours. _______________ 

_____ Informal training (e.g., reading manual) 

A) Was this training helpful?_____ Not helpful_____ Somewhat helpful_____ Very helpful_____ 

B) Please indicate the number of training hours. _______________ 

_____ Other (please write in your response): _____________________________________ 

A) Was this training helpful?_____ Not helpful_____ Somewhat helpful_____ Very helpful_____ 

B) Please indicate the number of training hours. _______________ 
B4

Each of the following statements should be answered in relation to various aspects of both the GPS unit (GX60) and the multi-function display unit (MX-20). Use the following scale to indicate your level of agreement 

with each statement. 

0 = Not Applicable/Did Not Use 

1 = Strongly Disagree 

2 = Somewhat Disagree 

3 = Neither Disagree nor Agree 

4 = Somewhat Agree 

5 = Strongly Agree 

GX-60 

Statement Startup Creating 

Flight Plan 

Editing 

Flight Plan 

En Route 

Navigation 

Responding 

to System 

Messages 

Accessing 

Airport 

Information 

15. The equipment is easy to use. 

16. The equipment operating 

procedures are easy to remember. 

17. The equipment performs the 

functions necessary for my flying 

operations. 

18. The equipment provides most of 

the information I need to conduct 

my flight. 

 

19. The equipment allows me to 

perform all necessary flight 

functions without reference to 

operating manuals, etc. 

 

20. The equipment provides easy 

access to the functions necessary 

for my flight. 

 

21. The equipment operating manual 

clearly explains procedures. 

 

22. When I press the wrong button, it is 

easy to undo. 

 

23. I am never confused about which 

display page is active. 

24. The equipment accommodates 

"canned" flight plans. 

25. The menu choices or button formats 

are desirable. 

26. The menus are easy to find. 
B5

MX-20 

Statement 

Accessing 

and Using 

Custom Map 

Function 

Accessing 

and Using 

Terrain 

Function 

Accessing 

and Using 

Traffic 

Function 

Accessing 

and Using 

Flight Plan 

Function 

Accessing 

and Using 

VFR Chart 

Function 

Accessing 

and Using 

IFR Chart 

Function 

27. The equipment is easy to use. 

28. The equipment operating 

procedures are easy to remember. 

 

29. 

The equipment performs the 

functions necessary for my flying 

operations. 

30. 

The equipment provides most of 

the information I need to conduct 

my flight. 

 

31. 

The equipment allows me to 

perform all necessary flight 

functions without reference to 

operating manuals, etc. 

 

32. 

The equipment provides easy 

access to the functions necessary 

for my flight. 

 

33. The equipment operating manual 

clearly explains procedures. 

 

34. When I press the wrong button, it 

is easy to undo. 

35. I am never confused about which 

display page is active. 

36. The PAN feature is desirable and 

easy to use. 

37. The INFO mode is very useful. 

38. The North-Up mode is preferred. 

39. The Track-Up mode is preferred. 

40. It's easy to adjust the map scale. 

41. The available map scales were 

appropriate to my flight tasks. 

 

42. The advisory flags (terrain or 

traffic) are easy to respond to. 

 

43. The 360 mode is preferred. 

44. The Arc mode is preferred. 

45. The INVERT function is 

desirable. 

 

46. The NAV data function is 

desirable. 

 

47. The range and display formats are 

desirable. 

 

48. Text message displays are 

appropriate and informative. 
B6

Statement GX-60 MX-20 

49. The response time of the equipment is adequate. 

50. The equipment provides adequate feedback. 

51. The display does not wash out in direct sunlight. 

52. The display does not wash out in indirect sunlight. 

53. The display is legible in night conditions. 

54. The controls are easy to operate. 

55. Control labels are easy to understand and remember. 

56. The equipment allows for easy detection of alerting messages. 

57. Alerting messages are appropriate and easily understood. 

58. The equipment-operating manual is easy to use. 

59. I feel confident using the equipment for VFR navigation. 

60. I feel confident using the equipment to aid in the visual acquisition of 

other aircraft. 

 

61. I feel confident using the equipment to aid in separation from terrain 

during VFR flight. 

 

62. The equipment is helpful as a supplemental system during IFR flight. 

63. The colors used to code the traffic symbols on the display were 

appropriate. 

 

64. The color coding used on the display was consistent with that of other 

flight deck displays. 

 

65. The colors used on the display aided decision-making. 

66. The symbology used on the display was consistent with that of other 

flight deck displays. 

 

67. The scale of the objects on the display was appropriate. 

68. I was distracted from other flying tasks when using the equipment. 

69. Overall, use of the equipment enhanced my decision-making. 

70. Overall, use of the equipment did not increase head-down time. 

71. Overall, use of the equipment did not increase workload. 

72. Overall, the equipment contributed to the successful completion of tasks. 

73. The physical layout of buttons allowed them to be used accurately. 

74. The size of the buttons made them easy to use. 

75. Button presses were accompanied by appropriate feedback. 

76. The location of the equipment allowed it to be seen easily. 

77. The reach required to use the equipment was acceptable from my seating 

position. 

 

78. The readability of the text on the equipment display was acceptable from 

my seating position. 

 

79. Having to read the text displays interfered with the performance of flight 

duties. 

 

80. The addition of the equipment in the cockpit decreased the time 

available to scan instruments. 

 
B7

Airport Surface Situation Awareness 

Please indicate your level of agreement to the following by blackening the circle in the column of your choice. 

Statement 

Not 

applicable/ 

Did not use

Strongly 

disagree 

Somewhat 

disagree 

Neither 

disagree 

nor agree 

Somewhat 

agree 

Strongly 

agree 

81. 

Overall, use of the MX-20 increased my ability 

to accurately locate other aircraft on the 

ground. 

     

82. 

Use of the MX-20 for airport surface situation 

awareness aided my ability to understand ATC 

communications. 

     

83. The relative position of other traffic was easy 

to interpret and understand on the MX-20.      

84. It was easy to get oriented to the surface map 

display.      

85. In general, enough information was provided 

on the MX-20 to be useful.      

86. 

In general, more information should be 

displayed on the MX-20 to aid in airport 

surface situation awareness. 

     

87. 

If you are in agreement with item 86, what additional information would be useful? ____________________ 

_______________________________________________________________________________________ 

88. The MX-20 display accurately showed my 

position on the ground (taxiway).      

89. The display symbol for my aircraft was easy to 

identify.      

90. 

If you are in disagreement with item 89, please explain why? ______________________________________ 

_______________________________________________________________________________________ 

91. The surface map was easy to bring up on the 

display.      

92. Use of the MX-20 during taxi increased the 

time available for crew duties.      

93. Use of the MX-20 during taxi increased the 

time available for completing checklists.      

94. Using MX-20 on the airport surface aided in 

locating targets visually.      

95. Use of the MX-20 aided in supporting traffic 

awareness while on the airport surface.      

96. Display clutter was not a problem during 

airport surface operations.      

97. Use of the MX-20 during airport surface 

operations did not increase head-down time.      

98. 

If in agreement with item 97, did using the MX-20 enhance or reduce the safety of airport surface operations? 

_______________________________________________________________________________________

_______________________________________________________________________________________ 

99. 

Do you feel the MX-20 helped, hindered, or had no effect on your overall performance during departure 

spacing operations? ______________________________________________________________________ 

_______________________________________________________________________________________ 
B8

Visual Acquisition 

Please indicate your level of agreement to the following by blackening the circle in the column of your choice. 

Statement 

Not 

applicable/ 

Did not use

Strongly 

disagree 

Somewhat 

disagree 

Neither 

disagree 

nor agree 

Somewhat 

agree 

Strongly 

agree 

108. Use of the MX-20 aided in visually acquiring 

traffic before receiving an ATC call.      

109. Use of the MX-20 aided in visually acquiring 

traffic after receiving an ATC call.      

110. 

ATC traffic, when visually acquired, appeared 

at the same clock position as depicted on the 

MX-20. 

     

111. ATC traffic, when visually acquired, appeared 

at the same distance as depicted on the MX-20.      

112. Use of the MX-20 had no effect on maintaining 

awareness of multiple traffic targets.      

113. Display clutter was not a problem when trying 

to acquire aircraft visually using the MX-20.      

114. The Select function was useful for visually 

acquiring traffic.      

115. Use of the MX-20 to enhance visual acquisition 

did not increase head-down time.      

116. 

If in agreement with item 116, did using the MX-20 enhance or reduce the safety of visual acquisition 

procedures?_____________________________________________________________________________

_______________________________________________________________________________________ 

117. 

When using the MX-20, were you distracted when other pilots used your call sign? If so, please explain. 

_______________________________________________________________________________________

_______________________________________________________________________________________ 

Departure Spacing 

Please indicate your level of agreement to the following by blackening the circle in the column of your choice. 

Statement 

Not 

applicable/ 

Did not use

Strongly 

disagree 

Somewhat 

disagree 

Neither 

disagree 

nor agree 

Somewhat 

agree 

Strongly 

agree 

100. Use of the MX-20 for departure/climb-out 

aided in traffic awareness.      

101. The MX-20 range settings were adjusted to 

increase traffic awareness during departure.      

102. Display clutter was not a problem during 

departure spacing operations.      

103. Use of the MX-20 during departure spacing 

operations did not increase head-down time.      

104. 

If in agreement with item 103, did using the MX-20 enhance or reduce the safety of departure spacing 

operations? _____________________________________________________________________________ 

_______________________________________________________________________________________ 

105. It was easy to transition from final approach to 

surface operations using the MX-20.      

106. 

The procedure for using the MX-20 to 

transition from final approach to surface 

operations was acceptable. 

     

107. 

Do you feel the MX-20 helped, hindered, or had no effect on your overall performance during departure 

spacing operations? _______________________________________________________________________ 

_______________________________________________________________________________________ 
B9

Final Approach Spacing 

Please indicate your level of agreement to the following by blackening the circle in the column of your choice. 

Statement 

Not 

applicable/ 

Did not use

Strongly 

disagree 

Somewhat 

disagree 

Neither 

disagree 

nor agree 

Somewhat 

agree 

Strongly 

agree 

118. 

The amount of effort required to keep the 

traffic to follow displayed on the MX-20 was 

acceptable. 

     

119. 

The amount of effort required to keep the 

traffic to follow displayed on the MX-20 was 

acceptable. 

     

120. Minimal effort was required to keep other 

traffic displayed on the MX-20.      

121. The Select feature was used frequently to ID 

the traffic to follow.      

122. The Select feature was used frequently to ID 

other traffic.      

123. 

The workload (including speed/power changes, 

etc.) necessary to achieve separation was 

acceptable. 

     

124. 

The workload (including speed/power changes, 

etc.) necessary to maintain separation was 

acceptable. 

     

125. 

When using the MX-20, closure to final 

spacing occurred at what seemed a comfortable 

and appropriate rate. 

     

126. The final spacing achieved when using the 

MX-20 was comfortable and appropriate.      

127. 

The workload for gauging distance behind the 

aircraft to follow was acceptable when using 

the MX-20. 

     

128. 

The Select feature provided enough 

information to maintain the distance behind the 

traffic to follow when using the MX-20. 

     

129. The MX-20 did not distract from crew duties 

during final approach operations.      

130. The MX-20 did not distract from checklist 

completion during final approach operations.      

131. The MX-20 did not increase head-down time 

during final approach operations.      

132. 

If in agreement with item 131, did using the MX-20 enhance or reduce the safety of final approach 

operations? _____________________________________________________________________________ 

_______________________________________________________________________________________ 

133. The MX-20 increased traffic awareness during 

final approach operations.      

134. 

Do you feel that the MX-20 helped, hindered, 

or had no effect on your overall performance 

during final approach operations? 

     

135. 

Do you have any additional comments regarding the use of the MX-20 for final approach operations? 

_______________________________________________________________________________________ 

_______________________________________________________________________________________ 
B10

Final Approach Runway Occupancy Awareness 

Please indicate your level of agreement to the following by blackening the circle in the column of your choice. 

Statement 

Not 

applicable/ 

Did not use

Strongly 

disagree 

Somewhat 

disagree 

Neither 

disagree 

nor agree 

Somewhat 

agree 

Strongly 

agree 

136. Use of the MX-20 effectively enhanced my 

awareness of final approach traffic.      

137. Use of the MX-20 effectively enhanced my 

awareness of departure traffic.      

138. 

Observing the target change color (e.g., brown 

to cyan) was helpful in determining the flight 

status of the target. 

     

139. 

When in take-off position, use of the MX-20 

enhanced my awareness of aircraft clearing the 

runway. 

     

140. 

When in take-off position, use of the MX-20 

enhanced my awareness of ground aircraft and 

vehicles on nearby taxiways. 

     

141. The MX-20 aided in gauging when the traffic 

to follow was over the runway threshold.      

142. The MX-20 aided in gauging when the traffic 

to follow was touching down.      

143. The MX-20 aided in gauging when the traffic 

to follow was clear of the runway.      

144. During low approaches, the MX-20 aided in 

detecting targets positioned on the runway.      

145. During low approaches, the MX-20 aided in 

detecting targets positioned near the runway.      

146. The MX-20 aided in detecting targets crossing 

the runway.      

Please indicate your response to the following statements by using the scale below. 

1 = Poor 

2 = Fair 

3 = Good 

4 = Very good 

5 = Excellent 

How would you rate the MX-20 overall as an aid to... 

147. ...airport surface situation awareness? ____________ 

148. ...departure spacing? __________________________ 

149. ...visual acquisition? __________________________ 

150. ...approach spacing? __________________________ 
B11

Comments/Suggestions

Each of the following statements should be answered in relation to various aspects of both 

the GPS unit (GX-60) and the multi-function display unit (MX-20). Comments will be 

summarized, so please print or write clearly and be concise. 

151. Do you have any comments regarding the use of color and symbology on the GX-60? 

152. Do you have any comments regarding the use of color and symbology on the MX-20? 

153. Do you have any additional comments regarding specific problems you have encountered 

with the GX-60? 

154. Do you have any additional comments regarding specific problems you have encountered 

with the MX-20? 

155. What recommendations would you make for improving the design of the GX-60? 

156. What recommendations would you make for improving the design of the MX-20? 

157. What recommendations would you make for improving the use of the GX-60? 

158. What recommendations would you make for improving the use of the MX-20? 

This information is NOT required but would be appreciated. If we require clarification of any of 

your responses, may we contact you? If your answer is yes, please provide the following 

information. 

 Phone: (______)_______________________ 

 

 E-mail: ______________________________ 

Thank you for your participation! 
B12

CHECKLISTS FORM 

G = Ground U = Upwind X = Crosswind D = Downwind B = Base F = Final 

Checklist Name (1) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Checklist Name (2) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Checklist Name (3) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Checklist Name (4) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Checklist Name (5) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Checklist Name (6) Status Discuss CDTI Effects ONLY 

Where completed (circle one): 

G U X D B F 

 Completed 

 Not Completed 

 Interrupted 

 Suspension 

 Missed by Crew 

 Missed by Observer 

 Other 

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ Aircraft Tail Number: N_______ Date: 10/______/00 

Circuit: ➀ ➁ ➂ ➃ ➄ ➅ Page: ➀ ➁ Flying Pilot:  Captain  FO 
B13

VISUAL ACQUISITION AND TRAFFIC AWARENESS FORM 

Traffic 1 Traffic 2 Traffic 3 Traffic 4 Traffic 5 Traffic 6 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

_______/______

_ 

_______/______

_ 

_______/______

_ 

_______/______

_ 

_______/______

_ 

_______/______

_ 

Order Order Order Order Order Order 

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄

Notes Notes Notes Notes Notes Notes 

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No

Call sign used ? 

 Yes  No

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No

Traffic 7 Traffic 8 Traffic 9 Traffic 10 Traffic 11 Traffic 12 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

Aircraft 

ID/Type 

_______/_______ _______/_______ _______/_______ _______/_______ _______/_______ _______/_______

Order Order Order Order Order Order 

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Visual ➀ ➁ ➂

CDTI ➀ ➁ ➂

ATC ➀ ➁ ➂

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄ ➀ ➁ ➂ ➃ ➄

Notes Notes Notes Notes Notes Notes 

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No

Call sign used ? 

 Yes  No

Call sign used ? 

 Yes  No 

Call sign used ? 

 Yes  No

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ Aircraft Tail Number: N_________ 

Date: 10/______/00 Circuit: ➀ ➁ ➂ ➃ ➄ ➅ Flying Pilot:  Captain  FO 
B14

MX-20 PLATFORM PROCEDURES FORM 

 

Airport Surface Situation Awareness (ASSA) 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...determine taxi sequence? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate local traffic situations? Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. ...locate targets relative to ground 

reference points (e.g., taxiways)? 

Y 

N 

N/A 

Y 

N 

N/A 

Y 

N 

N/A 

Y 

N 

N/A 

CDTI Features C1 C2 C3 C4 Notes 

1. Use Select to identify assigned taxi 

interval? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Final Approach Runway Occupancy Awareness (FAROA) while on GROUND 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate runway occupancy? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate final approach corridor for 

possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. ...evaluate departure corridor for 

possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Which map orientation was used most 

frequently? 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

2. Use In and Out to adjust range during 

taxi? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Departure Spacing 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...determine spacing? Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Use In and Out to adjust range prior 

to takeoff roll? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Use Traffic Altitude Filter to set 

altitude range filter for departure 

spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ Aircraft Tail Number: N_________ 

Date: 10/______/00 Flying Pilot:  Captain  FO 
B15

Initial Approach Spacing (Crosswind, Downwind and Base Legs) 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...establish and maintain spacing? Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Use Select to identify target? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Use Time (Vector lines) to aid in 

spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Use Traffic Altitude Filter to set 

altitude range filter for initial approach 

spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Final Approach Spacing 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...establish and maintain spacing? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate spacing by pilot not flying 

ONLY? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Use Select to identify target? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Set Time (Vector lines) to aid in 

spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 
B16

Final Approach Runway Occupancy Awareness (FAROA) while in AIR 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate the runway environment 

for possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Use In and Out to adjust range within 

runway environment? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Which map orientation was used most 

frequently? 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

3. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Runway Incursion Scenarios (Flight Periods 1 and 2 ONLY) Notes 

1. Was TRIOS1 Van identified? CDTI ➀ ➁ Visual ➀ ➁ No 

Distance ___________nm (check CDTI for distance) 

2. Was FAA 727 (N40) identified? CDTI ➀ ➁ Visual ➀ ➁ No 

Distance ___________nm (check CDTI for distance) 
B17

Surface Navigation 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate the airport surface 

environment for possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...identify taxiways on airport 

surface? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features C1 C2 C3 C4 Notes 

1. Use In and Out to adjust range on 

airport surface? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Which map orientation was used most 

frequently? 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up

3. Use Time (Vector lines) to evaluate 

target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Use Select to identify target? Y 

N 

Y 

N 

Y 

N 

Y 

N 

5. The Surface Map was used to taxi... 

 Never  Sometimes 

 Always  N/A 

Taxi Time Structured Routes ONLY Taxi Speed Notes 

(see dots on map) Location? Proximate AC ID? (If Available) 

AC Type 

___________ Outbound 

 Structured 

 Unstructured 

Start 

___________ 

Stop 

___________ 

 Correct 

 Incorrect AC ID 

___________

Taxi Speed 

___________ 

AC Type 

___________ Inbound 

 Structured 

 Unstructured

Start 

___________ 

Stop 

___________ 

 Correct 

 Incorrect AC ID 

___________

Taxi Speed 

___________ 
B18

CDTI BUILD 2/4 AND PROTOTYPE PROCEDURES FORMS 

 

Airport Surface Situation Awareness (ASSA) 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...determine taxi sequence? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate local traffic 

situations? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. ...locate targets relative to 

ground reference points (e.g., 

taxiways)? 

Y 

N 

N/A 

Y 

N 

N/A 

Y 

N 

N/A 

Y 

N 

N/A 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

1. Use TGT (SEL) to identify 

assigned taxi interval? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Use VEC to evaluate target 

speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Final Approach Runway Occupancy Awareness (FAROA) while on GROUND 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate runway occupancy? Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate final approach 

corridor for possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. ...evaluate departure corridor for 

possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

1. Which display mode was used 

most frequently? DSP (ARC)

Arc 

Comp 

Rose 

(No Arc/ 

No Rose)

Arc 

Comp 

Rose 

(No Arc/ 

No Rose)

Arc 

Comp 

Rose 

(No Arc/ 

No Rose)

Arc 

Comp 

Rose 

(No Arc/ 

No Rose)

2. Use R↑ and R↓ to adjust range 

during taxi? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Use VEC to evaluate target 

speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ Aircraft Tail Number: N_________ 

Date: 10/______/00 Flying Pilot:  Captain  FO 
B19

Departure Spacing (Events 1, 2, and 3 – Long 6.0 nm) (Events 4 and 5 – Short 4.5 nm) 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...determine separation distance? Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

2. Set RR to separation distance? 

(Long = 6.0, Short = 4.5) 

(Build 4 ONLY) 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Use R↑ and R↓ to adjust range 

prior to takeoff roll? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Set VEC to 2 min Long or 1.5 min 

Short? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

5. Use LVL (ALT) to set altitude 

range filter for departure spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Initial Approach Spacing (Events 1, 2, and 3 – Long 9.0 nm) (Events 4 and 5 – Short 7.0 nm)

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...establish and maintain separation 

distance? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

2. Set RR to separation distance? 

(Long = 9.0, Short = 7.0) 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Select TGT (SEL) to identify 

target? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Set VEC to 2 min Long or 1.5 min 

Short? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

5. Use LVL (ALT) to set altitude 

range filter for initial approach 

spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

6. Use VEC to evaluate target speed? Y 

N 

Y 

N 

Y 

N 

Y 

N 
B20

Final Approach Spacing (Events 1, 2, and 3 – Long 5.0 nm) (Events 4 and 5 – Short 3.0 nm) 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...establish and maintain separation 

distance? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...evaluate separation by pilot not 

flying ONLY? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

1. Set RR to separation distance? 

(Long = 5.0, Short = 3.0) 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Use TGT (SEL) to identify target? Y 

N 

Y 

N 

Y 

N 

Y 

N 

3. Set VEC to 2 min Long or 1.5 min 

Short? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Use VEC to evaluate target speed? Y 

N 

Y 

N 

Y 

N 

Y 

N 

Final Approach Runway Occupancy Awareness (FAROA) while in AIR 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate the runway environment 

for possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

1. Use R↑ and R↓ to adjust range 

within runway environment? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Which display mode was used most 

frequently? ARC (DSP)

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

3. Use VEC to evaluate target speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Runway Incursion Scenarios (Flight Periods 1 and 2 ONLY) Notes 

1. Was TRIOS1 Van identified? CDTI ➀ ➁ Visual ➀ ➁ No  

Distance ___________nm (check CDTI for distance) 

2. Was FAA 727 (N40) identified? CDTI ➀ ➁ Visual ➀ ➁ No  

Distance ___________nm (check CDTI for distance) 
B21

Surface Navigation 

CDTI Used to... C1 C2 C3 C4 Notes 

1. ...evaluate the airport surface 

environment for possible conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. ...identify taxiways on airport 

surface? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI Features (BLD2) C1 C2 C3 C4 Notes 

1. Use R↑ and R↓ to adjust range on 

airport surface? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. Which display mode was used most 

frequently? ARC (DSP)

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

Arc 

Comp 

Rose 

(No 

Arc/No 

Rose) 

3. Use VEC to evaluate target speed? Y 

N 

Y 

N 

Y 

N 

Y 

N 

4. Use TGT (SEL) to identify target? Y 

N 

Y 

N 

Y 

N 

Y 

N 

5. The Surface Map was used to 

taxi... 

 Never  Sometimes 

 Always  N/A 

Taxi Time Structured Routes ONLY Taxi Speed Notes 

(see dots on map) Location? Proximate AC ID? (If Available)

AC Type 

___________ Outbound 

 Structured 

 Unstructured 

Start 

___________ 

Stop 

___________ 

 Correct 

 Incorrect AC ID 

___________

Taxi Speed 

___________

AC Type 

___________ Inbound 

 Structured 

 Unstructured

Start 

___________ 

Stop 

___________ 

 Correct 

 Incorrect AC ID 

___________

Taxi Speed 

___________
B22

Modified Cooper-Harper Workload Scale 

Instructions: Begin in lower-left corner and follow arrows up and/or to the right. Darken the circles that correspond 

with the CAPTAIN’S role (PF or PNF) and his/her level of mental effort (1-10). 

Difficulty level Pilot demand level Rating 

Very easy, 

highly desirable 

Mental effort is minimal and 

desired CDTI performance is 

easily attainable 

1 

Easy, 

desirable 

Mental effort is low and 

desired CDTI performance is 

attainable 

2 

Fair, mild 

difficulty 

Mental effort is required to 

attain acceptable CDTI 

performance 

3 

Minor but 

annoying 

difficulty 

Moderately high mental effort 

is required to attain acceptable 

CDTI performance 

4 

Moderately 

objectionable 

difficulty 

High mental effort is required 

to attain acceptable CDTI 

performance 

5 

Very 

objectionable but 

tolerable difficulty 

Maximum mental effort is 

required to attain acceptable 

CDTI performance 

6 

Major difficulty 

Maximum mental effort is 

required to bring errors to 

moderate level 

7 

Major difficulty 

Maximum mental effort is 

required to avoid large or 

numerous errors 

8 

Major difficulty 

Intense mental effort is 

required for acceptable CDTI 

performance, but frequent or 

numerous errors persist 

9 

Impossible Acceptable CDTI performance 

cannot be accomplished reliably 10 

Post-Flight Structured Interview 

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ Aircraft Tail Number: N___________ Date: 10/______/2000 

Full Stop 1:  PF  PNF ➀ ➁ ➂ ➃ ➄ ➅ ➆ ➇ ➈ ➉

Full Stop 2:  PF  PNF ➀ ➁ ➂ ➃ ➄ ➅ ➆ ➇ ➈ ➉

Full Stop 3:  PF  PNF ➀ ➁ ➂ ➃ ➄ ➅ ➆ ➇ ➈ ➉

Full Stop 4:  PF  PNF ➀ ➁ ➂ ➃ ➄ ➅ ➆ ➇ ➈ ➉

Is the CDTI 

satisfactory 

without 

improvement? 

Mental 

workload is 

high and 

should be 

reduced 

Major 

deficiencies, 

CDTI redesign 

is strongly 

recommended

Major 

deficiencies, 

CDTI redesign 

is mandatory

Is adequate 

spacing 

performance 

attainable with 

tolerable 

workload? 

Can safe use of 

the CDTI for 

spacing be 

accomplished 

most of the 

time? 

Yes 

Yes 

Yes 

N

N

N

START 
B23

Please answer these questions as they relate to the following OpEval-2 applications: Departure Spacing [CDTI Build 

2 and 4], Approach Spacing, Airport Surface Situation Awareness, and Final Approach Runway Occupancy 

Awareness. 

1. As a crew, were you adequately prepared for today’s mission? 

2. In general, please discuss the effect of the MX-20/GX-60 equipment on your interaction with ATC. 

3. In general, please discuss the effect of the MX-20/GX-60 equipment on crew roles and communication (e.g., 

checklist completion). 

4. In your opinion, what rules or procedures need to be developed and/or implemented to support the efficient use 

of the MX-20 for OpEval-2 applications. 

5. Did you make a maneuver based on the CDTI only? If so, why? 

6. Are there any general or specific issues (e.g., display, control panel, etc) about the MX-20/GX-60 equipment 

that you would like to discuss? 

7. Would you like to see the MX-20/GX-60 equipment implemented in all aircraft that you fly? 

8. Were you comfortable with your knowledge of the MX-20/GX-60 system? 

9. Was there a tradeoff between the goal of increased traffic awareness for some other goal or task(s)? 

[CDTI Build 2 and 4 ONLY] 

10. What information should be available to ATC to support Departure Spacing using the CDTI? 

Thank you for your time and candid responses to our questions. 

Flight Event: ➀ ➁ ➂ ➃ ➄ ____ 

Aircraft Tail Number: N___________ 

Date: 10/______/2000