2000-01-5520

Initial Evaluation of CDTI/ADS-B for Commercial Carriers: CAA's Ohio
Valley Operational Evaluation

**Vernol Battiste and Rose Ashford**

**NASA Ames Research Center**

**Baltazar Oscar Olmos**

**MITRE Center for Advanced Aviation System Development**

Copyright © 2000 Society of Automotive Engineers, Inc.

**Abstract**

Flight activities during the Cargo Airline Association's Ohio Valley
Operations Evaluation (OpEval) were focused on near-term Cockpit Display
of Traffic Information (CDTI) applications. Seven CDTI applications were
ranked from highest to lowest priority, and the first two, Enhanced
Visual Acquisition for "See & Avoid", and Enhanced Visual Approaches,
were evaluated during OpEval. Five other applications were demonstrated.
For the Enhanced Visual Acquisition and Enhanced Visual Approach
applications, a detailed, comprehensive operational concept document was
prepared. The operational concept and the associated CDTI requirements
were tested during OpEval. Both pilots and controllers reported that the
CDTI augmented the visual acquisition and visual approach tasks and
improved pilot awareness of surrounding traffic. Additionally, the
results suggest operational performance benefits in the form of enhanced
spacing awareness and a potential reduction in the misidentification of
aircraft called out by ATC. No overriding human factors issues were
revealed that would negatively impact operational approval of these two
applications for traffic environments similar to OpEval. Flight crews
identified three issues, display integration, clutter, and head down
time, which need to be considered as we proceed with the design and use
of CDTI. One potential issue raised by the controllers that needs to be
addressed, is that of flight crews initiating unwarranted requests from
ATC.

INTRODUCTION

In July 1999, an initial evaluation of a prototype CDTI that depicts
surveillance information regarding surrounding traffic was conducted at
Airborne Airpark in Wilmington, Ohio. Twelve aircraft operated by three
member airlines of the Cargo Airline Association (CAA) flew an intensive
series of flight trials providing a basis for evaluating near-term CDTI
applications and data link technologies. The flight test was
accomplished in partnership with the FAA's Safe Flight 21 program, the
MITRE Center for Advanced Aviation System Development, NASA, DoD, and
other industry and academic partners \[1\].

The flight trials were focused on evaluation of near-term CDTI
applications that can be implemented without significant changes to ATC
procedures. These applications have been identified by RTCA and are
described in the Automated Dependent Surveillance-Broadcast (ADS-B)
Minimum Aviation System Performance Specification (MASPS), RTCA DO-242,
and the CDTI MOPS \[2,3\]. The flight scenarios were designed to enable
evaluation of flight crew and controller performance, operational
procedures and benefits, and data link technical performance for these
applications. The evaluation is intended to support operational approval
of the CDTI for these near-term applications, and be sufficiently
comprehensive that, although significant future effort will be required
for certification, there should be no unresolved "show stoppers."

This paper describes the human factors data collection program, together
with the ensuing analysis. Other accomplishments of OpEval include data
link evaluation, radar and ADS-B track analysis of aircraft spacing with
and without the CDTI, and ATC voice tape analysis of message frequency
and content. A complete description of OpEval and these results can be
found online at [www.faa.sf21.gov](http://www.faa.sf21.gov/) \[4\].

Much of the development work to design the flight scenarios was
conducted at the Integration and Interaction Laboratory (I-Lab) operated
by MITRE. The I-Lab is a medium-fidelity simulation facility including
both flight deck and ATC environments. It was used to develop specific
test scenarios and procedures for evaluating the CDTI. During the final
I-Lab simulation, the test scenarios were "flown", and the human factors
data collection process verified.

CDTI Applications Evaluated

Application priorities were (highest to lowest priority):\[5\]

-   Evaluate Enhanced Visual Acquisition for "See & Avoid"

-   Evaluate Enhanced Visual Approaches

-   Demonstrate Airport Surface Situation Awareness

-   Demonstrate Enhanced In-Trail (or lead) Climb/In-Trail Descent

-   Demonstrate Station Keeping

-   Demonstrate Departure Spacing

-   Demonstrate Final Approach Spacing

The Enhanced Visual Acquisition application was intended to aid flight
crews in visually acquiring proximate traffic, and in increasing overall
traffic awareness. Pilots using a CDTI will continue their normal visual
scan with an additional aid to where to focus their attention. The CDTI
serves as an enhancement for the visual acquisition of traffic,
including ground vehicles, thus improving safety and efficiency of
flight operations. A major expected benefit of the CDTI is
reacquisition, or maintaining acquisition of previously acquired
traffic. RTCA developed a detailed Operational Concept document for the
Enhanced Visual Acquisition application, including flight crew and
controller roles and procedures, and CDTI requirements. The operational
concept was used to design flight scenarios that would allow us to
evaluate the application's benefits, and assess flight crew and
controller performance and acceptance.

The Enhanced Visual Approach application was intended to augment a
normal visual approach by providing additional traffic information that
will allow the flight crew to determine traffic position,
identification, ground speed and track. The CDTI is expected to reduce
the probability of loss of visual contact, and aid judgment of closure
and encounter geometries. An Operational Concept document was similarly
developed for this application. The OpEval implementation of Enhanced
Visual Approaches also included two additional CDTI tasks beyond a
standard visual approach. First, when responding to an ATC traffic
call-out, the pilot could include the call sign of that traffic. In
addition, the flight crews, once cleared for the visual approach, were
tasked with visually closing-up spacing on the lead aircraft using the
CDTI. This is considered a VMC implementation of the final approach
spacing application.\[8\]

The lower priority applications were demonstrated to show feasibility
during OpEval, but without integration into a fully operational context,
as well as allowing for only limited data collection. In particular, the
Airport Surface Situational Awareness application was hampered by lack
of an airport map on the display, although data was obtained for this
application from all the participating aircraft.

CDTI

The prototype Cockpit Display of Traffic Information (CDTI) that was
used for OpEval is shown in Figure 1. The display primarily depicts the
position of proximate traffic with respect to ownship. Two sources of
traffic information were available at OpEval: ADS-B and Traffic
Information Service (TIS). ADS-B is a surveillance system whereby the
GPS position and altitude (obtained from an altitude encoder or air data
computer) of an airborne vehicle, and GPS position of ground vehicles,
is periodically blind broadcast, along with the sender's identification.
The broadcast may then be received by any vehicle or ground station,
allowing for determination of the sender's position relative to the
receiver. TIS is a ground-based data link service by which an aircraft's
position as determined by Mode S surveillance radar is re-broadcast to
the ownship's Mode S transponder and again may be used to determine the
aircraft's position relative to the ownship. Unlike ADS-B, TIS includes
an alerting algorithm in the event the aircraft is considered a threat
to the ownship. TIS tracks Mode A, C or S equipped aircraft within
approximately 55 nautical miles of the Mode S ground radar sites and
uplinks traffic within five nautical miles and +/- 1,200 feet of the
ownship and any other tracked traffic considered to be a threat.\[6,7\]

![](media/image1.wmf) Figure 1: UPS-AT CDTI and Control Panel

As shown in Figure 1, the ownship is depicted by a white triangle with
the actual location of the ownship at the apex. ADS-B 'targets' are
depicted by cyan chevrons, and TIS targets as cyan "bullets". Both
targets "point" in the direction the traffic is moving and are
accompanied by a data tag including relative or absolute altitude in
100's of feet, an ↑ or ↓ arrow if the target is climbing or descending
more than 500 feet per minute, and aircraft identification for ADS-B
targets. The CDTI was configurable, with ranges from 0.5 to 320 nautical
miles, and allowing for the ownship to be centered on the display or
three quarters of the way down from the top, and presented both with and
without the compass rose. A TIS traffic alert occurred when the ground
station determined the target to be a threat to the ownship. This is
depicted by the target turning yellow and flashing, together with the
voice alert "TIS traffic".

Keyboard Functions

The major functions of the keyboard are listed below. Some other
functions are available, but could not be evaluated at OpEval and are
not discussed here.

Altitude Range

Defines the altitude range in 100's of feet above and below ownship
where targets will be displayed. It is accessed through the menu "MNU"
key.

Map Range "R↑, R↓"

Adjusts the display range from 0.5, 1. 2. 5. 10, 20, 40, 80, 160 to 320
nautical miles.

Display Mode Control "ARC"

Cycles through display options with ownship centered on the display or
3/4 down from the top, and with or without the compass rose.

Altitude Key "ALT"

Toggles between relative altitude or pressure altitude on displayed
targets.

Vector Key "VEC"

Toggles between viewing and not viewing ground track vectors.

Select Key "SEL"

Highlights the nearest ADS-B target. Depressing the FR↑ key highlights
the next farther ADS-B target, while depressing the NR↓ key highlights
the next nearer ADS-B target. The selected target changes color from
cyan to green and an augmented data block is presented in the lower left
portion of the display.

Graphic Closure Indicator "GCI"

This is available on selected targets only and is accessed by toggling
the ← and → keys. When the selected target is moving away from the
ownship at 5 knots or greater, the GCI appears as a thickened bar on the
target's ground track vector. Conversely, if the target is converging on
the ownship by 5 knots or greater, the thickened bar appears behind the
target symbol. GCI's are not shown for relative speeds less than 5
knots.

De-clutter Key "DCL"

Toggles between viewing and not viewing the data tags, ground track
vectors and range ring.

![](media/image2.wmf){width="3.497916666666667in"
height="1.9368055555555554in"}

Figure 2: CDTI Display System Schematic.

Link Data Processor Unit (LDPU)

The primary function of the LDPU is to fuse surveillance data from TIS
and ADS-B so that only one target is presented on the CDTI, even though
surveillance information may have been received from more than one
source. The LDPU also receives the ownship GPS position for display on
the CDTI. Part of the purpose of OpEval was to evaluate the candidate
data links. Mode S 1080 MHz and Universal Access Transceiver (UAT) were
evaluated. Very High Frequency Data Link Mode 4 (VDLM4) was not
available.

Flight Scenarios

Four DC-9's, operated by Airborne Express, and eight Boeing 727's, four
each operated by United Parcel Service and Federal Express, participated
in the flight test and human factors data collection. Several other
aircraft including a Boeing 757 operated by NASA and a Navy P-3 Orion,
participated in OpEval, and provided CDTI "targets", but were not part
of the human factors data collection effort. The DC-9's were operated
with a normal two-person crew, consisting of captain and first officer.
The B-727's were operated with a normal three-person crew consisting of
captain, first and second officers. The CDTI was located forward of the
throttles on both aircraft, slightly towards the captain's side. Due to
the seating arrangement, the second officer in the B-727 was unable to
reach the CDTI keyboard, and his or her view of it may have been
restricted. The flight test was conducted at Airborne Airpark in
Wilmington, Ohio on July 10, 1999. The airfield consists of two parallel
runways with 3,400 feet separation and has an operating control tower.
Dayton Approach and Indianapolis Center controlled the surrounding
airspace.

Eight of the twelve participating aircraft were assigned to the "Low"
flight scenarios designed for evaluation of the two high priority
applications, Enhanced Visual Acquisition and Enhanced Visual
Approaches. These aircraft conducted multiple approaches to the two
parallel runways, four aircraft in the traffic pattern for each runway,
and each approach ending in a go-around initiated at about 200 feet AGL
(see Figure 3). Since not all of the twelve aircraft were equipped with
fully operating CDTI's, equipped aircraft were assigned first to the
"Low" flight scenarios to maximize data collection for the two high
priority applications to be evaluated. Human factors observers flew on
all eight of the "Low" flight scenarios to maximize data collection.

During the low flight scenarios, the flight crews contacted Dayton
Approach after departure. After radar contact, Dayton approach vectored
the aircraft back to the final approach course; providing vectors for
cross wind, downwind, base, and initial turn to final. When each flight
crew reported either their traffic to follow on final, or the airport,
in sight, they were cleared for a visual approach. Dayton Approach also
called out any relevant traffic during each approach. The flight crew's
task during each approach was to use the CDTI in visually acquiring
traffic in their vicinity, both as part of their normal scan and after
an ATC traffic call. They were also asked to utilize the CDTI to
evaluate the spacing between their aircraft and the traffic ahead, and
to reduce spacing to a distance that was comfortable and that would
support a safe landing behind the traffic.

To aid in assessing the impact of CDTI on visual approaches and approach
spacing, a data collection matrix was developed. Each flight crew was
expected to complete seven visual approaches over the course of the
flight period. The first approach was considered a familiarization
flight. For the remaining six approaches, two were flown without the aid
of the CDTI and four with the CDTI. The collection of baseline data was
either the first two or final two visual approaches, and was randomized
among flight crews.

Weather conditions at Wilmington did not permit visual approaches during
the morning flight period. However, visibility improved during the day,
and visual approaches were flown during the afternoon session. 91 visual
approaches were then flown, 23 Baseline (no CDTI) and 68 with the CDTI.
The initial plan had been to conduct 96 approaches, 32 Baseline and 64
CDTI, for the whole day. Since the visibility during the afternoon
remained pretty much constant, we were able to gather sufficient data
for meaningful analysis, although it is somewhat limited by the lack of
baseline data.

![](media/image3.wmf)**Figure 3:** Low Altitude Flight Profile.

For the "High" flight scenarios designed to demonstrate the lower
priority applications, one aircraft had an inoperative CDTI, and on
another, the CDTI was only partially operating. Both aircraft appeared
as targets for the remaining aircraft with operating CDTI's so we were
able to demonstrate the "High' flight applications, Station Keeping and
Enhanced In-Trail Climb/In-Trail Descent, and Lead Climb/Lead Descent.
Human factors observers flew on one of the fully equipped aircraft, and
pilot opinion data was collected from another equipped aircraft, so we
were able to collect sufficient data for demonstration purposes. FAA
observers also flew on two of these aircraft, and although they were not
part of the human factors data collection effort, individual opinions
from those observers are reported in this paper.

The flight crew's task during both the station keeping and the in-trail
climb and descent was to maintain 15 +/- 1 nautical mile in trail
spacing behind the traffic they were following during the "round robin"
flight from Wilmington to Wilmington (see Figure 4). Flight crews were
not expected to visually acquire the traffic, but to select the traffic
on the CDTI, and to utilize the information provided on the CDTI display
to maintain the appropriate distance. Crews only performed the spacing
task during the straight segments of the flight, not during turns.

Both "Low" and "High" flight aircraft participated in data collection
for Airport Surface Situational Awareness while taxiing on the airport
surface.

![](media/image4.wmf){width="3.498611111111111in"
height="2.2680555555555557in"}

Figure 4: High Altitude Flight Profile.

Human Factors Evaluation

The objective of the human factors evaluation was to determine flight
crew and controller performance with, and acceptance of, the CDTI. The
observers recorded visual acquisition time and visual acquisition
performance both with and without the CDTI. Questionnaires were
administered to both flight crews and controllers after the flight test
was completed and structured debrief sessions were conducted. The flight
crew questionnaire included CDTI feature preferences and elicited
opinions on the usefulness of the CDTI for the evaluated and
demonstrated applications as well as workload and other issues. The ATC
questionnaire included controller experience with handling CDTI-equipped
aircraft.

Flight Crew Human Factors Observers

Human factors observers were selected and trained to record the flight
crew's performance with and without the CDTI during the OpEval flights

Observer training was conducted in the following areas:

-   The role of CDTI in the air traffic environment

-   The objectives and goals of the operational evaluation

-   UPS Aviation Technologies CDTI (features, operational capability)

-   The structure of visual approaches at an air carrier hub operation

-   The prescribed crew and ATC roles and procedures, during visual
    approaches and en-route cruise flight

-   Video presentation of type specific aircraft cockpits and procedures
    (B-727 for FedEX, UPS, and DC-9 for Airborne Express) for visual
    approaches and en-route cruise flight

-   The flight deck data collection procedures for low altitude and en
    route flights

```{=html}
<!-- -->
```
-   Use of data collection apparatus and procedures (flight deck and
    debrief/questionnaire)

    The observers took notes on specially designed observer data
    collection forms, administered the post-flight questionnaire, and
    debriefed the flight crews after the test scenarios were completed.
    The observer records were used to support the collection of:

```{=html}
<!-- -->
```
-   Response time to each traffic call

-   Assessment of CDTI use during visual traffic acquisition

-   Assessment of the impact of CDTI on normal cockpit duties

The observer protocol for collecting visual acquisition performance data
on the flight deck was to code as an acquisition any recognition of
traffic in sight by the crew; through inter-cockpit verbal or non-verbal
communications, or verbal reports to ATC. Flight deck observers were
asked to encode the visual acquisition performance (strategies) into
five categories. The five categories were: (1) visual only no CDTI, (2)
visual first then CDTI, (3) CDTI first then visual, (4) both - order
unknown, and (5) CDTI only no visual. The first category is descriptive
of the currently used visual acquisition strategy, where the flight crew
searches the visual scene for known and unknown traffic, with or without
an ATC traffic call without the aid of a traffic display. The second
strategy supports the use of the traffic display to confirm a visual
sighting of traffic. The third approach suggests that the flight crews
utilize the traffic display to support the visual search; locating the
traffic on the display, then searching the visual scene for the traffic.
The fourth category allows the observer to code the acquisition when
they are not able to determine the order, visual or CDTI first, used by
the flight crew. The fifth category suggests a method of maintaining
general situational awareness of traffic not in sight, but in the
general area and is not recorded as a visual acquisition.

Training: Flight Crews and Controllers

Flight crews received training in the purpose, limitations and
operations of a CDTI the morning of the day before OpEval. They were
briefed on the flight scenarios and other OpEval requirements (safety,
contingency planning, etc.) that afternoon.

Controllers were asked to conduct operations as normal (i.e., no
specific training was required). However, they were asked to point out
all relevant traffic and to vary the order in which each aircraft
conducted the visual approach. This was done to prevent each aircraft
from following the same aircraft on every approach. Two of the three
controllers who participated in OpEval also participated in the I-Lab
simulations.

Flight Crew Post Event Questionnaire

Pilot opinion ratings were gathered during the post-flight debriefing,
after each crew completed their post flight duty requirements. The
questionnaires were completed prior to a structured interview and the
combined activity lasted about one hour. Each question was designed to
elicit specific information from the flight crews on the usefulness of
individual features and functions of the CDTI, and the impact of the
CDTI on specific flight-related tasks. Items in the questionnaire were
scaled from 1 to 3 or from 1 to 5 to support a Likert scale analysis.
Additionally, a selection of "Not Applicable/Did Not Use" was an
available choice.

For specific features of the display, crews were asked to rate the ease
of use of each individual feature, from easy to use, OK to use and
difficult to use. These ratings were translated into 1 -- easy, 2 -- OK,
and 3 difficult to support the comparison. The neutral point of 2.0 was
used as an anchor to evaluate all 1 to 3 ratings and those found to be
significantly ([p]{.underline}\<0.05) above or below 2.0 on a two-tailed
t-test were indicated as a positive or negative response by the group on
that specific feature or function.

Additionally, flight crews were asked to provide opinions on the impact
of the CDTI on specific flight related tasks, particularity, the impact
of information presented on the CDTI during visual traffic acquisition,
visual approach, departure spacing, in-trail and lead climbs and
descents, and station-keeping. Questions related to these tasks were
presented in a 1-5 Likert scale format with the lower portion of the
scale having negative statements (strongly disagree), the upper portion
having positive statements (strongly agree) and the middle of the scale
having a neutral statement (neither agree or disagree).

A final set of questions in the questionnaire asked the flight crews to
summarize their overall experiences with and without the CDTI
information during OpEval. The anchor for the lower portion of the
summary questions was "excellent", "poor" for the upper portion, and
"OK" for the middle.

Crew responses to each question were translated into numeric values from
1 to 5, or 1 to 3 and analyzed using a two-tailed t-test. Flight crew
opinions were compared to the neutral response for each question and
those found to be significantly (p\<0.05) above or below the neutral
point are reported as a positive or negative response by the group.

I-LAB Simulations

The objectives of the I-Lab simulations were to develop the
application-specific procedures and scenarios that were flown during the
OpEval flights. The intent was to test and validate all OpEval
activities in the I-Lab before the actual flights at Wilmington. The
I-Lab simulations included both the flight deck and ATC environments.
The I-Lab simulations were also used to support prototype design and
development activities. Additional objectives of the I-lab simulations
were to assist in the development and testing of the data collection
process, and to familiarize observers and controllers with developed
procedures. The I-Lab simulation environment is composed of a simulator
cockpit, a computer-generated visual scene, and controller station. The
Wilmington Ohio flight environment (e.g., navigation aids, visual scene)
was modeled in support of this simulation.

Flight Deck

The I-Lab cockpit simulation is configured to approximate the
performance of a twin engine, transport category airplane. The cockpit
simulation is a fixed based, glass cockpit coupled to a projection
visual system that provides a view of the external visual scene. The
flight dynamics and performance approximate those of the Boeing 757. The
system uses side stick controls coupled with an autopilot to control
vertical and lateral flight. Thrust control is accomplished through a
pilot-selectable auto throttle system, or manually through the thrust
levers. Cockpit displays are software generated and are modeled to be
similar in form and function to the displays installed on the Boeing
747-400. The Electronic Flight Instrument System (EFIS) displays include
a Primary Flight Display (PFD) of attitude, airspeed, altitude and
vertical rate information for basic aircraft control, and a
mode-selectable Nav Display (ND) which depicts lateral navigation
information in a plan view map. The CAA CDTI information and related
display features are presented on the ND using the Map mode. The CAA
CDTI control panel is located on the center console between the two
pilots.

Visual Scene

The visual scene encompasses a 150^0^ lateral by 40^0^ vertical field of
view using a single screen front-view projection system with a refresh
rate of 30 Hz. Targets appearing on the traffic display are correlated
with visible traffic in the out-the-window view. The terrain for the
visual scene was built using the Defense Mapping Agency's (DMA) Digital
Terrain Elevation Data (DTED) and accuracy was ensured through the use
of the National Oceanic and Aerospace Administration (NOAA) airport
obstruction charts.

ATC

The cockpit is linked to a controller station that is composed of a
combined TRACON and Tower position. ATC communications are accomplished
using headsets and microphones in the simulator. The pseudo pilots
positions are provided with a voice party line to simulate ATC and pilot
communications of other aircraft.

Results

This section summarizes the flight deck and controller data accumulated
during the OpEval process, which includes the initial I-Lab OpEval
simulations. Inferences and conclusions drawn on this data are discussed
in the final section of the paper. Additionally, summaries of flight
reports from other observers are included, two from the FAA and one from
a Navy P-3, all flying the "High" flight scenarios. These are not part
of the structured data collection effort.

I-Lab OpEval Simulation Results

The primary objective of the I-Lab simulation activity was to develop
the application-specific procedures, scenarios, and briefing materials
that were to be conducted at OpEval. The intent was to "fly" the OpEval
scenarios in the I-Lab before the actual flights at Wilmington.
Secondary objectives of the I-Lab simulation were to assist in the
development and testing of the data collection process and to
familiarize flight crews, OpEval controllers, and observers with the
scenarios. The following is a summary of the outputs from the I-Lab
simulations:

-   Finalized low and high altitude flight profiles

-   Familiarized Dayton controllers and subset of CAA crews with OpEval
    flight profiles

-   Developed flight procedures and phraseology (i.e., flight maneuver
    cards)

-   Finalized Human Factors observational data collection protocol

-   Developed Flight Crew Mission Guide

Enhanced Visual Acquisition

The CDTI enhanced visual acquisition application is a capability that
was expected to aid pilots in visually acquiring other proximate traffic
in the out the window (OTW) view as well as increasing their traffic
awareness. Pilots using a CDTI were expected to continue their visual
scan but would have an additional aid to visually acquire other aircraft
by focusing their attention to a specific area. This method of acquiring
visual traffic is expected to reduce the visual search time. The CDTI
would serve as an enhancement for the visual acquisition of traffic
including ground vehicles, thus improving the safety and efficiency of
flight operations. During OpEval, visual acquisition data was collected
throughout all flight maneuvers in the traffic pattern and during
departure/arrival phases. A major expected benefit of Enhanced Visual
Acquisition is reacquisition, or maintaining acquisition of previously
acquired traffic.

![](media/image5.wmf)

Visual Acquisition Performance- Flight Deck Observations

![](media/image6.wmf) in the following figures**.** Figure 5 shows that
observers recorded a total of 106 ATC traffic calls, and that 71 of
those calls were confirmed by the flight crew as traffic sighting. In
total, flight crews reported about 67% of ATC traffic calls in sight.
The percentages were approximately the same for the baseline/no CDTI and
CDTI conditions.

Figure 6 shows that prior to an ATC call, flight crews utilized a
variety of strategies during visual traffic acquisition. To acquire
traffic in the visual scene prior to an ATC traffic call, crews
referenced the CDTI then located the traffic in the OTW visual scene 37%
of the time, followed by visual search only and visual plus CDTI 24% of
the time each, and CDTI no visual 9% of the time. Figure 7 suggests that
after an ATC traffic call, flight crewmembers utilized the standard
method of searching the OTW visual scene for locating traffic 46% of the
time, followed by CDTI plus visual search 26% of the time, visual plus
CDTI 19%, and CDTI no visual 7% of the time.

![](media/image7.wmf)

Flight Crew Questionnaire and Debrief Data (Visual Acquisition)

Flight crew members rated the CDTI as aiding them significantly in
visually acquiring traffic, before and after an ATC traffic call (mean
rating of 4.8 and 4.9

respectively on a scale of 1 to 5, (p\<.05, df 23). They also reported
that maintaining an awareness of multiple traffic targets was less
difficult when using the CDTI (mean rating with CDTI, 4.8, p\<.05 df 23;
without CDTI, mean rating of 1.25, p\<.05, df 23). Flight crew members
agreed that there was a good match between the physical location of
traffic, ATC reported traffic position, and CDTI traffic position (mean
ratings of 4.8 and 4.8 respectively, p\<.05, df 23). Crews also rated
the "Select" function as useful for visual traffic acquisition and for
maintaining an awareness of traffic (mean rating 4.4, p\<.05, df 20).
However, three flight crew members, 2 in the high altitude group (DC-9)
and 1 captain (B-727) in the low altitude group reported not using the
select function.

Flight crews did not rate display clutter as being a problem during
visual acquisition (mean rating 3.04, p\> .05, df 23). They did rate
head down time as being a problem during visual traffic acquisition
(mean rating 3.625, p\< .05, df 23). During the debrief, when asked if
the CDTI helped or hindered visual acquisition performance, flight crews
generally confirmed their earlier rating that the CDTI was an aid. One
flight crewmember suggested that when using the CDTI to acquire traffic,
"you know exactly where to look." Crews also commented during the
debriefing that:

-   During visual approaches (visibility \> 5 miles) CDTI aided visual
    acquisition, especially for small aircraft.

-   During ILS approaches (visibility \< 5 miles) the CDTI increased
    flight crew confidence in their ability to maintain an awareness of
    the exact position of traffic, even when traffic transitioned in and
    out of obscurations.

-   Having the aircraft category and position information on the CDTI
    helped in verifying ATC traffic calls, and that this type of
    information also aided in identifying traffic prior to an ATC
    traffic call.

-   ![](media/image8.wmf)

-   The CDTI aided in planning and workload management, intra-cockpit
    communication, and maintaining an awareness of multiple aircraft
    during marginal visual meteorological conditions.

-   Their ability to utilize the CDTI in support of visual acquisition
    improved with experience and that unlike the TCAS traffic display,
    the traffic symbology on the CDTI provides instantly discernible
    heading information.

-   Head down time and clutter were problems and that the display needs
    to be integrated into existing displays to prevent task saturation.

Flight Crew Workload

Flight crews were asked to rate the ease of use for each CDTI setting
that was used during visual traffic acquisition. Overall, flight crews
reported that each function was easy to use (value of 1 = easy, 2 = OK,
3 = difficult, Table 1). The table also shows that some features (ALT
and GCI) were used by less than half of the flight crewmembers.

  ----------------- ------------- ------------- ------------- -------------
  Feature           t (all        df            mean          n
                    p\<.05)                                   

  Altitude Range    -4.27         16            1.35          26

  Map Range         -10.11        19            1.15          26

  ARC               -10.11        19            1.15          26

  ALT               -2.84         11            1.33          26

  VEC               -6.28         14            1.13          26

  SEL               -3.55         18            1.42          26

  GCI               -2.40         12            1.46          26

  G DCL             -5.44         16            1.23          26
  ----------------- ------------- ------------- ------------- -------------

Table 1: Flight Crew Workload for CDTI Setting during Visual Acquisition

The table shows, in the degrees of freedom column, the number of flight
crews (df+2) that used each function. Second officers on the B-727
aircraft normally reported not using each function, however this was not
always the case. The difference between "n" (total in the sample) minus
the df plus 2 equals the number of flight crew members that did not
utilize each function (n-(df+2)=not used/not applicable).

Evaluate Enhanced Visual Approaches

In response to the post-flight questionnaire, a majority of flight crews
agreed that the CDTI aided in overall traffic awareness (mean rating
4.88 compared to a neutral value of 3, p\<.05 df 23), and in closing to
a final approach spacing, which was comfortable and appropriate (mean
rating 4.54, p\<.05, df 23). Crews also agreed that the selected target
feature of the CDTI provided enough information (mean rating 4.50,
p\<.05, df 23) and that the workload for gauging the distance behind the
lead aircraft was acceptable (mean rating 4.58, p\<.05, df 23). Flight
crews did not find the workload acceptable for gauging the distance
behind the lead aircraft on final without the CDTI (mean rating 2.65,
p\>.05, df 22). Flight crews rated the CDTI as aiding crew duties during
visual approaches (mean rating 4.29, p\<.05, df 23). Flight crews rated
as neutral or mixed the use of the CDTI to: estimate when the lead
aircraft was over the landing threshold (mean rating 3.17, p\>.05, df
22); estimate when the lead aircraft was touching down (mean rating
3.59, p\>.05, df 21); when lead traffic was clearing the runway (mean
rating 3.14, p\>.05, df 20); or that the CDTI aided in completing
checklist (mean rating 2.61, p\>.05, df 22). Flight crews rated that
using the CDTI during visual approach increased head-down time (mean
3.58, p\<.05, df 23).

In the post-flight debrief, crews suggested that they currently do not
try to gauge exact distances, but try to maintain the spacing interval
that exists when ATC clears them for the approach. Flight crews also
agreed that using the CDTI during visual approaches increased head down
time, but when asked if the CDTI helped or hindered operations during
visual approach, the majority said that it helped. The following is a
summary of crew comments on how the CDTI helped during visual
approaches:

-   Allowed us to tighten up our approach

-   Very useful for acquiring and re-acquisition of traffic

-   Display of ground speed and distance information reduced the
    workload of following traffic

-   Increased situational awareness in busy visual traffic pattern

-   Supported re-checking the position of traffic without consulting ATC

-   Improved our awareness of ATC traffic pattern objectives

-   Using the system to support flight deck objectives improved with
    experience - for example, our confidence in maintaining a desired
    interval during the approach.

Flight crews also reported some issues with the current system. Crews
reported that the presentation of TIS information produced false and/or
fused targets, and occasionally data tags appeared to swap. Crews
reported that head down time was a problem and suggested that it was
easy to become fixated on the CDTI. Flight crews reported that the
location of the display, not in their primary visual scan, made it
difficult to integrate the CDTI into their normal scan pattern. And due
to the placement of the display in the cockpit, additional intra-cockpit
communication was required. Finally, crews reported that they could not
determine when a lead aircraft was over the landing threshold, but could
determine when the aircraft touched down. Flight crews suggested that to
perform other more precise tasks (e.g., determining when traffic was
over the landing threshold), an airport surface map would be required. A
number of the issues identified by flight crews were known and accepted
due to the fact that the initial prototype was developed to support
visual traffic acquisition, and general situational awareness, not
traffic alerts and ground/airport surface operations.

Flight Crew Workload

Flight crews were asked to rate the ease of use of each CDTI setting
that was used to visually acquire traffic during visual approaches.
Table shows that for the flight crews that used each function, they
rated each as easy to use. The table also shows the number of flight
crews (df+2) that used and did not use (n-(df+2)each feature. Second
officers on the B-727 aircraft normally reported not using each
function, however this was not always the case.

  --------------------- ------------ ------------ ------------ ------------
  Feature               t (all       df           mean         n
                        p\<.05)                                

  Altitude Range        -5.79        15           1.19         24

  Map Range             -10.1        19           1.15         24

  ARC                   -10.1        19           1.15         24

  ALT                   -2.84        11           1.33         24

  VEC                   -6.28        14           1.13         24

  SEL                   -4.98        18           1.32         25

  GCI                   -3.56        10           1.27         23

  DCL                   -4.01        17           1.33         24
  --------------------- ------------ ------------ ------------ ------------

Table 4: Flight Crew Workload for CDTI Setting During Visual Approaches.

Demonstrate Airport Surface Situation Awareness

This application of CDTI enables flight crews to observe surface traffic
positions on a real-time display and, along with any available visual
cues and radio communications, infer intent with respect to surface or
airborne movements. While on the airport surface, the CDTI would be used
to increase situational awareness in the flight deck by supplementing
visual acquisition, identification, and tracking. For OpEval, the
"see-and-avoid" procedure was still the primary means of conflict
avoidance. Even though both high and low flight aircraft were asked to
comment on the use of the CDTI to support surface situational awareness,
data collected for this application are regarded as for demonstration
only due to lack of a surface map. In order to demonstrate surface
situational awareness, all ADS-B equipped aircraft and ground vehicles
left their transponders on while on the airport surface to provide
targets.

Flight Crew Questionnaire and Debrief Data

Flight crews rated the CDTI as aiding in supporting traffic awareness
during airport surface operations (on a scale of 1 to 5, mean rating
3.8, p\<.05, df=24). The system was also rated as aiding in locating
ground traffic (mean 3.7, p\<.05, df =23), maintaining an awareness of
airborne traffic (mean rating 4.4, p \<.05, df=25), and distinguishing
between ground and airborne traffic (mean rating 4.0, p\<.05, df=23).

Flight crews rated the system as not significantly enhancing their
awareness of aircraft clearing the runway when in they were in the
takeoff position (mean rating 3.44, p\>.05,df=14), or their awareness of
traffic and vehicles on parallel taxiways (mean rating 3.2,
p\>.05,df=22). They also reported that while operating on the surface,
the CDTI did not significantly aid in locating traffic visually (mean
rating 3.54, P\>.05, df 26). Flight crew opinions were mixed on the
impact of CDTI on head down time (mean rating 3.2, p\>.05, df=26),
however they agreed that display clutter was a problem during surface
operations (mean rating 3.6, p\<.05, df=25). Flight crews rated the use
of the CDTI as not increasing the time available for other crew duties
(mean rating 2.5, p\<.05, df=25), nor for performing flight crew check
list (mean rating 2.5, p\<.05, df=25).

Flight Crew Workload

Flight crews were asked to rate the ease of use of each CDTI setting
used to enhance airport surface situation awareness, and with only one
exception, Graphic Closure Indicator, they rated each feature easy to
use. Table 5 provides mean crew ratings for each CDTI feature.

  ----------------------- ----------- ----------- ----------- -----------
  Features                t           df          mean        n

  Altitude Range          -5.41       17          1.26        25

  Map Range               -9.27       20          1.18        26

  ARC                     -7.3        20          1.18        26

  ALT                     -3.65       14          1.31        24

  VEC                     -5.78       18          1.25        25

  Select key              -5.55       19          1.29        26

  GCI                     -1.21       14          1.75        26

  DCL                     -3.74       18          1.4         26
  ----------------------- ----------- ----------- ----------- -----------

Table 5: Flight Crew Workload for Airport Surface Situation Awareness.

Demonstrate Departure Spacing

Although there was no requirement to maintain any prescribed spacing
during the departure phase of the high altitude mission, flight crews
reported that they used the CDTI to support general traffic awareness
and to maintain an awareness of the other aircraft conducting the high
altitude mission.

Flight Crew Questionnaire and Debrief Data

Flight crews rated the CDTI as aiding in supporting traffic awareness
during departure/climb (on 1-5 scale, mean rating 4.6, p\<0.05, df=4).
The crews adjusted the CDTI range settings to increase traffic awareness
(mean rating 4.75, p\<0.05,df=3).

Flight Crew Workload

Flight crewmembers were asked to rate the ease of use of the following
CDTI settings, used during departure/climb-out. Table 6 shows that for
those that used the feature, the feature was rated as easy or OK to use.
The table also shows the number of crewmembers that used or did not use
each feature.

  ----------------- ------------ ------------ --------------- ------------
  **Feature**       **Easy to    **OK to      **Difficult to  **Not used**
                    use**        use**        use**           

  Altitude Range    2            2            0               1

  Map Range         3            1            0               1

  ARC               4            0            0               1

  ALT               1            0            0               4

  VEC               1            1            0               3

  SEL               2            1            1               1

  GCI               0            0            1               4

  DCL               2            1            1               1
  ----------------- ------------ ------------ --------------- ------------

Table 6: Flight Crew Responses for CDTI Settings during
departure/climb-out.

Demonstrate Station Keeping

OpEval Flight crews used the CDTI equipment in order to safely conduct
Instrument Flight Rules (IFR) in-trail constant spacing for extended
periods of time. For the OpEval, all station-keeping maneuvers were
performed in a radar environment, with radar separation being maintained
throughout the maneuvers. The spacing criterion for these maneuvers was
15 nautical miles (nmi) +/-1 nmi which emulates new oceanic separation
standards, but is well beyond the required minimum radar separation of 5
nmi. No new ATC procedures were evaluated, but the performed maneuvers
and the associated collected data will support development of procedures
for both radar and non-radar en route airspace.

Results of Flight Crew Questionnaire and Debrief Data

Five crews were sampled on the following items. They rated the workload
required to achieve (on 1-5 scale, mean rating 4.6, n=5), and maintain
15+/- 1 nautical mile during station-keeping as acceptable, and that
minimal effort was required to keep the traffic to follow (mean rating
4.2, n=5), and other traffic (mean rating 4.0, n=5) displayed on the
CDTI. They rated that the CDTI aided in determining the spacing from the
traffic to follow during station-keeping (mean rating 4.8, n=5), and
that using the CDTI for station keeping aided in supporting traffic
awareness (mean rating 4.6, n=5). Of the crewmembers sampled, three
reported that the select target feature was used to ID the traffic to
follow (mean rating 4.3, n=3), and to ID other traffic (mean rating 4.3,
n=3). The other two crewmembers reported not using the select target
feature. Three flight crewmembers reported using the GCI, and rated that
using the feature to gauge the selected closure/separation trends was
easy (mean rating 2.7) According to their rating, the GCI did not
provide sufficient detail about the selected traffic closure/separation
trends (mean rating 2.3). The other two crewmembers sampled reported not
using the GCI.

Flight Crew Workload

Flight crewmembers were asked to rate the ease of use of the following
CDTI settings, used during the station-keeping task. Table 7 shows that
for those individuals that used each feature, a majority reported the
features to be easy or OK to use. The table also shows the number of
crewmembers that used or did not use each feature.

  -------------- -------------- -------------- -------------- --------------
  **Feature**    **Easy to      **Ok to use**  **Difficult to **Not used**
                 use**                         use**          

  Altitude Range 2              0              1              2

  Map Range      3              1              0              1

  ARC            3              1              0              1

  ALT            1              0              0              4

  VEC            1              1              0              3

  SEL            1              2              0              2

  GCI            0              0              1              4

  DCL            2              1              1              1
  -------------- -------------- -------------- -------------- --------------

Table 7: Flight Crew Responses for Station Keeping.

Demonstrate Enhanced In-Trail (or Lead) Climbs/Descents (ITC/ITD)

The CDTI enhanced ITC and ITD procedures are designed to allow a
trailing aircraft to climb or descend through a leading aircraft's
altitude. The CDTI will be used for distance and closure rate
determinations and the trailing aircraft will then execute an ITC or
ITD. During OpEval, aircraft were performing these maneuvers within a
previously coordinated block of airspace so pilots performed these
climbs/descents at their own discretion (i.e., contact with ATC is not
required). As was discussed with the station-keeping application, the
OpEval en route maneuvers were performed in a radar environment, with
radar separation being maintained at all times. The ITC/ITD maneuvers
were also performed at 15 nmi. In addition to performing the ITC or ITD,
the flight crew was also tasked to simultaneously maintain 15 nmi.
Again, no specific new ATC procedures were evaluated (e.g., no ITC
checklist, phraseology, or communication with lead aircraft), but the
performed maneuvers and associated collected data will support
development of procedures for both the radar and non-radar en route
airspace.

Flight Crew Questionnaire and Debrief Data

Five crewmembers (two, DC-9, and three, B-727) participated in the post
flight data collection effort for in-trail lead climbs and descents.
When asked, a majority of flight crews (4 of 5) reported "no"
differences in the usefulness of the CDTI settings for maintaining
separation during an in-trail climb or descent. Flight crews rated as
minimum the effort required to keep the traffic to follow displayed on
the CDTI during climb or descent (on 1-5 scale, mean rating 4.4). One
flight crew, three of the five pilots that completed the questionnaire,
rated the select target feature as useful to ID the traffic to follow
(mean rating 4.3, n=3), and to ID other traffic (mean rating 4.3, n=3).
Crewmembers did not rate the usefulness of the GCI as high in gauging
closure/separation trends during climbs and descents (mean rating 3.3,
n=3). One flight crew reported not using the select target or GCI
features during this phase of the mission. The flight crews rated as
acceptable the workload to achieve and maintain separation for station
keeping (15 +/- 1 nmi) during climbs and descents. (mean rating 4.2,
n=5; and mean rating 4.2, n=5, respectively). The crewmembers rated the
CDTI as aiding in supporting traffic awareness for station keeping
during climbs and descents (mean rating 4.2, n=5).

Flight Crew Workload

Flight crews were asked to rate the ease of use for each CDTI setting
that was used to assist in maintaining separation during in-trail climbs
and descents. For those flight crews that used each function, Table 8
shows crew ratings for each feature. The Second officer on the B-727,
who rated each feature as not used may not have had access to the
control panel.

  -------------- -------------- -------------- -------------- --------------
  Feature        Easy to use    Ok to use      Difficult to   Not used
                                               use            

  Altitude Range 2              0              1              2

  Map Range      2              2              0              1

  ARC            2              2              0              1

  ALT            1              1              0              3

  VEC            0              0              0              5

  SEL            0              2              0              3

  GCI            0              0              1              4

  DCL            2              0              1              2
  -------------- -------------- -------------- -------------- --------------

Table 8: Flight Crew Responses for CDTI Settings During In-Trail
Climbs/Descents.

Flight Crew Response to CDTI Display Ergonomics

CDTI Color and Symbology

Crews rated the color-coding used on the CDTI as appropriate (mean
rating 4.04, p\<.05, df=26) and consistent with other flight deck
displays (mean rating 3.70, p\<.05, df 26). They rated as helpful the
unique traffic symbols used to identify ADS-B and TIS traffic (mean
rating 4.0, p\<.05, df=24). However, two crews, using the first
generation displays, reported the color used to code ground traffic was
a problem. They suggested that the color brown was difficult to see on
the display, and that the use of brown for ground traffic should be
revisited.

CDTI Features

The flight crews rated the Flight ID (FID) data tags as easy to
understand (mean rating 4.45, p\<0.05, df=21), and the additional
information provided in the selected target data block (e.g., range,
ground speed) as also useful and easy to understand (mean rating 4.72,
p\<0.05, df=24). Crews rated the altitude information presented on the
display as easy to understand (mean rating 4.48, p\<0.05, df=23) and 19
of the 26 flight crews surveyed reported a preference for the relative
altitude format. This presentation is consistent with the TCAS
presentation of altitude information. When crews were asked about their
use of the graphic closure indicator, the majority rated the ground
speed information in the selected aircraft data block as more useful for
understanding closure rate (mean rating 4.58, p\<0.05, df=18). Flight
crews were mixed or neutral when asked whether when the ground track
vectors are displayed (VT key) and aircraft are in a turn, that the
actual flight path and the displayed track are different (mean rating
2.86, p\>0.05, df=20), which they are. Flight crews were also mixed or
neutral on the impact (positive or negative) on their performance of the
capability to independently select the ground track vector time and the
range map scale (mean rating 2.83, p\>0.05, df=17).

CDTI Control Panel

Flight crew members rated the CDTI control panel as easy to use (mean
rating 3.65, p\<0.05, df=22), and rated inputs that required button
cycling as easy to make (mean rating 3.78, p\<0.05, df=22). They rated
the labels on the keys as clearly identifying the key's function (mean
rating 3.75, p\<0.05, df=23), but some crew members suggested that the
keyboard was too complex. Crews rated the keys on the keyboard to be
appropriately spaced (mean rating 3.78, p\<0.05,df=22) and sized for
accurate and comfortable use (mean rating 3.74, p\<0.05, df=22), and
that feedback was adequate (mean rating 3.64, p\<0.05, df=22), but that
sensitivity and feedback could be improved.

CDTI Location and Readability

Flight crew members rated as adequate the symbols used for ownship (mean
rating 3.92, p\<0.05,df=25) and other ADS-B traffic (mean rating 4.15,
p\<0.05, df=25), as well as the readability of text (mean rating 4.15,
p\<0.05, df=25). Flight crews were mixed or neutral on the adverse
impact of ambient light on the resolution of the CDTI screen (mean
rating 2.73, p\>0.05, df=25). Flight crews were mixed or neutral that
the reach required to access the CDTI from their seat was acceptable
(mean rating 3.05, p\>0.05, df=21). Crew opinions were mixed when asked
if visual access to the information on the CDTI was equivalent to other
flight deck displays (mean rating 3.48, p\>0.05, df=24), 14 reported
that it was, while 12 reported that it was not. When asked if the CDTI
control panel was conveniently located for use crews ratings were mixed
(mean rating 3.38, p\>0.05, df=23), 14 crew members reported that it
was, while 13 reported that it was not. When asked if the viewing angle
to discern information on the CDTI was equivalent to other displays the
crew ratings again were mixed(mean rating 3.28, p\>0.05,df=24), 14
reported that it was, while 13 reported that it was not.

Transfer of Training between TCAS and CDTI

Flight Crews were asked to comment on their ability to transfer skills
and knowledge from their experience with TCAS to their use of a CDTI
system. The majority of flight crews reported that there would be a
positive transfer between the two systems and that the transfer should
be relatively easy as they gained more experience with the system.
Flight crews suggested that the transition between the two systems
should be transparent, and that the CDTI improved situational awareness
by providing traffic direction, call sign, aircraft category, and speed.
Flight crews suggested that the information provided by the CDTI better
supported visual traffic acquisition, however, they also suggested a
need for the traffic collision avoidance information provided by TCAS.

Questionnaire CDTI System Summary

A majority of crewmembers agreed that overall the CDTI System was an aid
to: (1) high altitude departure flight (on a scale of 1 to 3, mean 1.6);
(2) station keeping (mean rating 1.2); (3) in-trail or lead climbs and
descents (mean 1.4); (4) visual approach (mean 1.4, p\<.05, df=24); and
(5) visual acquisition (mean 1.4, p\<.05, df=23). Crews ratings were
mixed on the use of CDTI as an aid to surface awareness (mean rating
2.88, p\>.05, df=24).

ATC Response to CDTI

This section briefly describes how the participating controllers
perceived CDTI to affect their work. The controller questionnaire
contained questions about how CDTI affected five controller job factors:

1.  Maintaining a safe and efficient traffic flow

2.  Maintaining attention and situation awareness

3.  Prioritizing

4.  Providing control information

5.  Communicating.

Maintaining a safe and effective traffic flow

None of the controllers gave any negative ratings (the lowest was 4) or
wrote any negative comments in support of their ratings. The three
participating controllers' responses to the first question indicated
that CDTI had a positive effect on maintaining a safe and efficient
traffic flow during OpEval. Each wrote a supporting comment:

-   After a pilot used the wrong call sign for traffic to follow, the
    controller corrected the error.

-   On two successive departures, the controller called traffic to
    follow on the crosswind leg, and the aircraft followed the traffic
    without incident.

-   The controller reported having to issue less speed information to
    pilots after issuing their visual approach clearances.

A related question asked the extent to which (if any) the actions of
pilots with CDTI facilitated their normal air traffic control
activities. The controllers' responses indicated that their activities
were moderately facilitated; also, the controllers found the difficulty
of OpEval nearly the same as routine control operations for similar
numbers and types of aircraft and weather.

Maintaining attention and situation awareness

The controllers indicated that CDTI had very little effect on
maintaining their (the controllers') attention and situation awareness.
The only comment was that the controller's situation awareness was the
same with CDTI as without it. A related question was how the pilot's use
of traffic call signs obtained from the CDTI affected the controllers'
situation awareness. The controllers responded that their situation
awareness was somewhat improved.

Prioritizing

The controllers indicated that CDTI had a slight positive effect on
providing control information. One comment was that the use of CDTI
allowed the controller to call traffic earlier than normal, thus making
better use of the controller's time.

Providing Control Information

The controllers indicated that the use of CDTI had a positive effect on
providing control information. The three supporting comments stated
that:

-   CDTI improved the aircrews' situation awareness, so the controllers
    felt more certain that they were following the correct traffic (two
    controllers made this comment).

-   Pilots using CDTI were better able to maintain their own spacing.

Communicating

The controllers indicated that CDTI had a moderately positive effect on
communicating. Two controllers noted that pilot use of the call sign of
traffic to follow was beneficial and increased their certainty that the
aircraft was following the correct traffic.

Flight Reports

During the High flight profiles, several FAA observers were present on
two CAA aircraft: Their observations regarding the OpEval flights were
documented in two flight reports, which are not included in the human
factors data analysis above, but which are summarized below.
Additionally, a Navy P-3 flight crew participated and their flight
report is also presented below.

FAA Flight Report (1)

The first report documented the following list of issues which this
observer suggested should be addressed in future CDTI implementations:

-   The ground clutter for the ADS-B targets was excessive and
    completely obliterated the airport environment. A mechanism needs to
    be developed to minimize the ground clutter while in flight. The
    following suggestions were offered in the report to help minimize
    display clutter:

```{=html}
<!-- -->
```
-   The "GND" indication that was displayed with the surface targets
    could be removed.

-   Flight ID's of surface targets could be removed while airborne and
    displayed when weight is on wheels.

-   The size of the surface targets could be reduced while airborne and
    then returned to normal size when weight is on wheels.

-   The designated landing runway could be displayed on the CDTI and
    aircraft that are potential surface incursions could be highlighted
    on the display.

```{=html}
<!-- -->
```
-   The TIS limitations and deficiencies need to be corrected and
    enhanced. Three TIS self-alerts were observed during the OpEval
    flight. The report stated that while the TIS tracks did provide an
    increased degree of overall traffic situation awareness, the large
    number of nuisance/false TIS alerts may reduce some of that benefit
    and should be minimized.

-   In some instances, there were erroneous ground/air indications of
    surrounding traffic. That is, aircraft on the surface were sometimes
    displayed as airborne and vice versa. Enhancements should be
    provided to minimize these occurrences.

-   The CDTI display on one of the DC--9 aircraft had several display
    anomalies (e.g., TEST button not functional, SEL function
    inoperative) and a general recommendation was made to gather data
    during an In-Service Evaluation to begin identifying CDTI
    reliability.

-   Some of the flight crews had questions regarding the operation of
    the CDTI that were not fully addressed in the CDTI Pilot Guide. As a
    result, the pilot's guide should be reviewed and modified to help
    clarify flight crew uncertainties and questions that were raised
    during OpEval.

FAA Flight Report (2)

The second flight report documented the following issues, some of which
were also identified in Flight Report 1. The observer also suggest that
these issues should be addressed in future CDTI implementations:

-   The use of aircraft call signs may require additional training and
    formalization of communication protocols and should be considered as
    part of future flight evaluations.

-   The TIS limitations and deficiencies need to be corrected and
    enhanced. Two TIS false alerts were observed during the OpEval
    flight.

-   Symbology inconsistencies were observed and should be addressed in
    future implementations. Specifically, the NAV information on the
    CDTI was not consistent with the ICAO waypoint naming convention and
    system.

-   Some of the flight crews had questions regarding the operation of
    the CDTI (e.g., function of left/right arrow keys) that were not
    fully addressed in the CDTI Pilot Guide. As a result, the pilot's
    guide should be reviewed and modified to help clarify flight crew
    uncertainties and questions that were raised during OpEval.

-   Procedures should be developed to optimize pilot use of the CDTI.
    The following areas were mentioned in the report:

```{=html}
<!-- -->
```
-   Recommendations are needed on when it is appropriate to use which
    mode (e.g., ARC mode vs. compass rose).

-   General guidance on the appropriate ground track vector length
    (e.g., 30 seconds, 1 minute) based on the flight phase would be
    helpful.

-   Develop procedures for station keeping for both straight and level
    and turning portions (e.g., crossing a VOR) of the application.

NAVY P-3 Flight Report

NAVAIRSYSCOM's (PMA-209) Communications, Navigation, and
Surveillance/Air Traffic Management Integrated Product Team requested
that NAWCAD Patuxent River integrate the Automatic Dependent
Surveillance Broadcast (ADS-B) system into an aircraft to support a
flight demonstration of this new technology. The test aircraft was a
NP-3C, BuNo 158204. The flight demonstration, held on 10 July 1999, was
coordinated and led by the Cargo Airlines Association with support from
the Federal Aviation Administration. Total P-3 test time was 2 hr of
flight testing during which a cooperative target, an FAA Convair 560,
was used as truth data in evaluating the Cockpit Display of Traffic
Information (CDTI). Flight test was conducted in the airspace
surrounding the Airborne Express facility at Wilmington Airfield.

The ADS-B suite was composed of a Mode S data link transponder, a VHF
data link Mode-4 transponder, a CDTI display and associated keypad
Control Panel, and a Link and Display Processor Unit (LDPU) containing
embedded dual 1090 MHz receivers, dual Universal Access Transceivers,
and a GPS receiver. A Navigation Avionics Platform Integration Emulator
(NAPIE) provided own aircraft pressure altitude and heading data to the
CDTI by interfacing with the P-3 avionics. The CDTI and Control Panel
were located in the cockpit, and the remaining equipment was located on
a rack in the P-3 cabin. The P-3 ground/air control relay provided the
weight-on-wheels discrete which was necessary because aircraft on the
ground sent out a different squitter than airborne aircraft, and the
CDTI depicted ground traffic in brown and airborne traffic in blue. The
landing gear position discrete, provided by an operator-controlled
switch at the project rack for this test, triggered the CDTI to display
ground targets only when landing gear was extended. The LDPU recorded
its internally generated own aircraft position information along with
target information collected on each separate data link. The NAPIE
digitally recorded own aircraft position, altitude, airspeed and heading
information on a Jaz data storage drive. Additionally, a GP-KS102 video
camera and Hi 8mm recorder were installed in the cockpit and navigation
station to record the CDTI and to capture associated Interior
Communications System (ICS) and radio transmissions.

The main focus of this test was a Human Factors evaluation of the
CDTI-to-pilot interface. The methods used in the Human Factors
evaluation of the ADS-B system included direct observation during the
flight, post-flight debriefing, post-flight questionnaire, and analysis
of flight test video and audiotapes. Post-flight analysis of own
aircraft and target aircraft heading, position, and ground speed
revealed that these CDTI parameters were relatively accurate.

The ADS-B system performed very well in the stated objective of helping
pilots achieve improved situational awareness and in increasing visual
acquisition of other aircraft while in flight. Aircrew did not feel that
the CDTI improved their situational awareness on the ground due to the
fact that the display was extremely cluttered and many of the displayed
targets overlapped each other.

Overall, the pilots found the CDTI very intuitive and the majority of
the functions performed as described in the CDTI Pilot's Guide. The
aircrew found setting up the CDTI to be relatively easy, with the
exception of the keypads on the Control Panel, which were too small for
gloved fingers, and the changing of altitude display limits which was
rather cumbersome due to the number of keypad entries necessary. Since
the CDTI accurately displayed the location of own-ship and target
aircraft, the pilots were comfortable in using the CDTI for in-trail
separation and station-keeping. Targets were depicted in the correct
colors and with the features described in the CDTI Pilot's Guide. The
most serious issue found was that, on several occasions, the CDTI
incorrectly placed an ADS-B target on the CDTI and displayed it for
several seconds. The United Parcel Service Aviation Technologies was
investigating this anomaly at the time this report was written. The CDTI
was configured to request Traffic Information Services data for the
flight, but the data were intermittent, inaccurate, and inconsistent
with the ADS-B targets displayed.

An operational concern of the pilots was that while display of flight
ID's was very useful, it also creates the possibility of confusion and
distraction if either ATC or pilots start to use the other aircraft's
call sign in calling out traffic. In addition, with the wealth of
information displayed on the CDTI, pilots may tend to anticipate ATC
instructions and take on the role of the controller. Finally, pilots
need to be aware that not every aircraft is equipped with ADS-B and will
not be displayed on the CDTI.

Generalization of the Results

The data collected during OpEval are from to one day of flying with 12
aircraft and caution needs to be exercised in extrapolating the results
to more general situations. The major limitations on the data result
from the test scenarios not representing line flight operations (e.g.,
multiple approaches in place of a single approach after cruise flight,
and go-arounds instead of full stop landings), and that the CAA's CDTI
implementation is not representative of future implementations,
integrated with other flight displays. Controllers were also asked to
help evaluate the CDTI by calling out relevant traffic as much as
possible. This may have impacted some results, especially analysis of
voice communications.

Before OpEval, both pilots and controllers expressed concern over the
potential for confusion resulting from one flight crew using the call
sign of another aircraft. Pilots are highly attuned to their own call
sign, and might assume that any transmission including their call sign
was for them. Resolving this confusion might add to the pilot's
workload, to frequency congestion, and to controller workload. From the
controller's perspective, they might not be sure if the call was from
the call sign aircraft or from another aircraft about the identified
aircraft, again increasing frequency congestion and workload while the
confusion was resolved. Augmenting these concerns were reports of those
situations actually occurring during some of the I-Lab simulations. As a
result of the simulation work, communications procedures were developed
for OpEval to minimize call sign confusion. Again, these procedures may
limit generalization of results if final procedures are different from
those used at OpEval.

CONCLUSIONS AND IMPLICATIONS

OpEval provided a unique opportunity to evaluate the CDTI in an
operational flight environment. A considerable amount of flying time was
achieved, allowing for the collection of a large quantity of subjective
data. Both flight crews and controllers were very willing to share their
experience and opinions after the event, resulting in a wealth of
opinion data which has provided valuable insight into the benefits and
issues surrounding the use of CDTI for the evaluated and demonstrated
applications. Collectively, a review of the data revealed no
"showstoppers" that would indicate serious obstacles towards the
implementation of CDTI for the applications evaluated and demonstrated
at OpEval. Comments from both flight crews and controllers were
generally positive, although the data did not always support their
positive opinions. Overall, flight crews agreed that the CDTI aided
visual acquisition, visual approaches, station keeping, in-trail
climbs/descents, and high-altitude departure flights.

The human factors observer reports of visual acquisition performance
with and without the CDTI are consistent with the flight crew's
assertions that the CDTI was an aid to visual acquisition (Figures 6 and
7). Figure 6 shows that in the absence of an ATC traffic call, flight
crews acquired traffic 76% of the time using the CDTI, either before,
after, or without acquiring the traffic OTW. Only 24% of the time was
traffic acquired without the aid of the CDTI. After an ATC traffic call
(Figure 7), almost half the responses are OTW visual acquisitions only,
but CDTI was still used for acquisition in the remaining responses,
including 33% of the total responses where traffic is first acquired on
the CDTI. This suggests that while many responses to an ATC call are
traditional OTW visual acquisitions, the CDTI is still a significant aid
in that process.

Flight crews identified three issues, which need to be considered as we
proceed with the design and use of CDTI to support visual traffic
acquisition and other ADS-B applications. Crews identified display
integration, clutter, and head down time as issues that need to be
addressed in future CDTI implementations.

Flight crews reported that the location of the display, outside the
primary visual scan, made it difficult to integrate into their normal
scan, and that this location may have caused additional intra-cockpit
communication. Intuitively, integrating the CDTI with the NAV display in
a glass-cockpit aircraft should improve CDTI usability and reduce
head-down time; however, this remains to be demonstrated. Regardless,
flight crews reported the present CDTI implementation to be effective as
an aid to visual acquisition, either with or without an ATC traffic
call, and that maintaining awareness of multiple traffic targets was
less difficult with the CDTI. This would suggest that the CAA's initial
CDTI implementation, on a stand-alone display, was adequate for both the
enhanced visual acquisition and enhanced visual approach applications.

Some crews did identify issues with the location of the CDTI. It was
difficult for the second officer in B-727 aircraft to see and use the
CDTI which was located forward of the throttles, and he or she could not
reach it without leaving his or her seat. The DC-9 First Officers also
had less access to the CDTI, which was located nearer to the Captain's
side. The impact of the placement of the display will depend on flight
crew procedures for operation of the CDTI. Overall, the display location
required flight crews to develop alternatives to their usual cockpit
scan to include the CDTI and make use of the information being
presented.

In general, display clutter was reported to be manageable during
airborne operations, even in the relatively densely populated low flight
scenarios where aircraft were conducting visual approaches. Display
clutter was, however, especially evident during airport surface
operations, where a large number of targets were located in close
proximity. The combination of large target and data tag size, and no
airport surface map, contributed to a number of adverse remarks about
the CDTI's usability on the airport surface. Since the CDTI was not
designed for use on the airport surface, these adverse comments are not
surprising. A surface map will be added to the CDTI and evaluated in
future Operational Evaluations.

Many flight crews commented on the increase in head down time while
using the CDTI, while at the same time suggesting it was an effective
aid to visual acquisition and visual approaches, both currently out the
window tasks. One possible explanation for this reported increase in
heads down time is that flight crews were relatively inexperienced with
the CDTI and received only moderate instruction in its operation. There
was, however, some evidence that flight crews' confidence in and
efficiency with the CDTI improved over the course of the day. This would
suggest that the increased head down time may be mitigated with training
and experience.

Overall, flight crew response to CDTI color, symbology and features was
positive. Flight crews reported all CDTI settings used during visual
acquisition and visual approaches "easy to use" and all functions except
GCI "easy to use". The statistical analysis shows that the responses
overall were closer to "easy to use" than "OK to use", and that the
difference between the "easy to use" response and the "OK to use"
response was statistically significant. Half or fewer of those
responding for all applications except surface situational awareness
used the ALT and GCI features. Second Officers in the B-727 generally
reported not using all functions. These results indicate that, while the
keyboard functions may not be optimal, flight crews were able to use the
CDTI effectively even in high workload phases of flight.

Acknowledgments

The authors would like to express their appreciation to all those
individuals involved in the data collection efforts during OpEval. The
authors would also like to acknowledge the AATT Program for supporting
the OpEval CDTI/ADS-B evaluation.

References

1.  RTCA Free Flight Select Committee issued the "Joint
    Government/Industry Roadmap for Free Flight Operational
    Enhancements" in August 1998.

2.  RTCA. (1998). Minimum Aviation System Performance Standards (MASPS)
    for Automatic Dependent Surveillance Broadcast (ADS-B), DO242,
    Washington, D.C.

3.  RTCA (1999). Guidance for Initial Implementation of Cockpit Display
    of Traffic Information (CDTI). DO-243, Washington, D.C.

4.  FAA (2000). SafeFlight 21 Master plan. Washington, D.C.

5.  Olmos, B. O., Mundra, A. D., Cieplak, J. J., Domino, D. A., and
    Stassen, H. P. (1998). Evaluation of near-term Applications for
    ADS-B/CDTI Implementation. Proceedings of SAE/AIAA World Aviation
    Congress, Warrendale, PA: Society for Automotive Engineers, Inc.

6.  Horvath, S., Damschem, P., and Schulte, C. (1998). ADS-B CDTI Pilot
    Guide -- PD1257. Cargo Airline Association, Washington, D.C.

7.  RTCA (In Press) Minimum Operational Performance Standards for
    Cockpit Display of traffic info. Washington, D.C.

8.  Abbott, Terrence S. (1985). A Compensatory Algorithm for the
    Slow-Down Effect on Constant-Time-Separation Approaches. NASA
    Technical Paper 2386, Langley, VA.