PILOT ACCEPTANCE, COMPLIANCE, AND PERFORMANCE

WITH AN AIRBORNE CONFLICT MANAGEMENT TOOLSET

Nathan A. Doble Richard Barhydt Dr. Karthik Krishnamurthy

Titan Corporation NASA Langley Research Center Titan Corporation

Hampton, VA Hampton, VA Hampton, VA

A human-in-the-loop experiment was conducted at the NASA Ames and
Langley Research Centers, investigating the En Route Free Maneuvering
component of a future air traffic management concept termed Distributed
Air/Ground Traffic Management (DAG-TM). NASA Langley test subject pilots
used the Autonomous Operations Planner (AOP) airborne toolset to detect
and resolve traffic conflicts, interacting with subject pilots and air
traffic controllers at NASA Ames. Experimental results are presented,
focusing on conflict resolution maneuver choices, AOP resolution
guidance acceptability, and performance metrics. Based on these results,
suggestions are made to further improve the AOP interface and
functionality.

Introduction

In today's air transportation business environment, aircraft operators
are increasingly looking for means to increase flight efficiency.
However, with air travel demand once again rising to levels that
exacerbate delays and challenge the capacity of the National Airspace
System (FAA, 2004), large efficiency improvements may be difficult to
realize under current operational conditions. As a result, it has been
acknowledged that a transformational, rather than evolutionary, approach
to air traffic management modernization is needed (DOT, 2004).

As part of the Advanced Air Transportation Technologies project, NASA
has developed such a far-term, transformational concept, called
Distributed Air/Ground Traffic Management (DAG-TM) (NASA, 1999). The
goals of DAG-TM are to increase efficiency and maintain safety through a
redistribution of decision-making authority among airborne and
ground-based elements of the air transportation system. It is a
gate-to-gate concept, addressing all flight phases from dispatch to
arrival.

En Route Free Maneuvering

En Route Free Maneuvering is one component of DAG-TM, addressing the en
route and terminal-transition phases of flight. In an En Route Free
Maneuvering environment, trained crews of equipped aircraft assume
responsibility for traffic separation. Such crews would be free to
modify their flight path in real time, without approval from an air
traffic controller, as long as basic flow management initiatives are
complied with (e.g., crossing a terminal airspace entry point at a
specified time). These flights would operate under a new set of flight
rules called Autonomous Flight Rules (AFR).

Except for busy terminal areas, where AFR operations would not be
permitted, AFR traffic would be integrated with Instrument Flight Rules
(IFR) traffic. AFR flight crews would be responsible for separation from
both IFR and other AFR aircraft. Air traffic controllers would issue
flow management constraints to all aircraft, and continue to provide
separation among IFR aircraft, accommodating those operators who choose
not to equip for AFR. By distributing separation assurance among
multiple airborne and ground-based elements in this way, the National
Airspace System may be able to absorb a higher increase in demand beyond
what is possible with a centralized, ground-based approach.

Background

Previous Research

The work presented in this paper builds upon previous studies conducted
at NASA as well as initial Free Flight research by organizations such as
NLR in the Netherlands (Hoekstra et al., 2000). Past NASA experiments
investigated such topics as AFR operations in confined airspace and the
use of aircraft intent for decision making (Krishnamurthy et al., 2003).

The Autonomous Operations Planner

Central to AFR operations are the capabilities of airborne conflict
prevention, detection, and resolution, as well as adherence to traffic
flow management constraints. It is assumed that pilots cannot safely
perform these functions without some form of decision support. As such,
NASA Langley Research Center has developed a prototype airborne toolset
called the Autonomous Operations Planner (AOP) (Barhydt & Krishnamurthy,
2004).

The prototype AOP interface is designed around a modern "glass cockpit"
flight deck. It provides conflict alerts and resolution guidance via the
navigation display, using state and intent data from the ownship and
proximate traffic. To meet flow constraints, it also generates
conflict-free paths that achieve Required Times of Arrival (RTAs) at
waypoints. The AOP has been developed using a human-centered approach,
with resolution guidance complementing the pilot's choice of control
mode. For example, when the aircraft is being flown in a tactical mode
(e.g., a selected heading or altitude) or when very near-term conflicts
exist, resolution guidance is presented as a simple heading or vertical
speed command. When the aircraft is flown in a strategic mode (i.e.,
coupled to the aircraft's flight management system (FMS)), resolution
guidance is presented as an FMS route modification.

Conflicts are displayed by highlighting the intruder aircraft and
indicating the region of conflict along the active flight path with a
colored "dog bone." The AOP also provides information to help pilots
avoid inadvertently creating new conflicts while maneuvering. These
conflict prevention tools take on two forms: Maneuver Restriction Bands
and Provisional Conflict Alerts. Maneuver Restriction Bands are
displayed as "no fly" heading and vertical speed ranges. Using a "dashed
dog bone" symbology, Provisional Conflict Alerts show regions of
conflict along *proposed* flight paths (e.g., a modified but unexecuted
FMS route or a selected but unengaged heading). Figure 1 shows an
example of AOP symbology on a Boeing 777-style navigation display.

![](media/image1.wmf){width="3.004861111111111in"
height="3.0756944444444443in"}

Figure 1. AOP Interface with Strategic Resolution

Experimental Approach

In summer 2004, the NASA Ames and Langley Research Centers jointly
conducted a human-in-the-loop simulation of En Route Free Maneuvering
operations (Barhydt & Kopardekar, 2005). This experiment extended the
previous research in several ways. A realistic, mixed AFR-IFR operating
environment was simulated, including overflight aircraft as well as
arrivals. The AOP was enhanced to provide vertical resolution guidance
in addition to lateral guidance. In addition, interactions with
ground-based air traffic controllers were studied.

This paper presents a subset of the En Route Free Maneuvering
experimental results, focusing on conflict resolution maneuver choices,
pilot-reported acceptability of AOP guidance, and performance metrics,
including how pilot compliance with AOP affected resolution performance.

Participants

Test subjects included 12 pilots at NASA Langley as well as pilots and
air traffic controllers at NASA Ames. The NASA Langley subject pilots
were all Airline Transport Pilot rated with experience in Boeing glass
cockpit aircraft. These pilots flew workstation-based flight simulators
that emulated the displays of an AOP-equipped Boeing 777. Additional AFR
and IFR background traffic was supplied with pseudo-pilot stations
staffed by research personnel.

Figure 2 shows the experimental airspace. It consisted of simulated
high- and low-altitude sectors of a portion of Fort Worth Center. The
sectors were staffed at NASA Ames by five FAA-qualified air traffic
controllers. They provided separation services between IFR aircraft and
were given automated tools for conflict detection and resolution. In
addition, researchers acted as pseudo-controllers in large "ghost"
sectors surrounding the experimental sectors, providing limited services
to flights entering and exiting the subject-controlled airspace.

![](media/image2.wmf){width="3.0659722222222223in"
height="1.7055555555555555in"}

Figure 2. Experimental Airspace

Scenario Design

The experiment was designed in a within-subjects format, with 16
different scenarios. Four different traffic conditions were simulated,
which varied the amount of traffic as well as the relative proportions
of AFR and IFR overflight aircraft. Table 1 details the four traffic
conditions.

Table 1. Traffic Conditions Tested

  ---------------- -------------------------------- ---------- -----------
  **Condition**    **Avg. Traffic Density**         **% IFR**  **% AFR**

  C1               slightly above current Monitor   100%       0%
                   Alert parameter                             

  C2               equal to C1 density              75%        25%

  C3               ≈1.5 × C1 density                50%        50%

  C4               ≈2 × C1 density                  35%        65%
  ---------------- -------------------------------- ---------- -----------

At each of the four traffic conditions, pilots flew two overflight
profiles and two arrival profiles. Except for C1 scenarios (in which all
flights were IFR), subject pilots were responsible for resolving
scripted and unscripted traffic conflicts. AOP alerted pilots to
conflicts up to 10 minutes prior to predicted Loss of Separation (LOS).
Pilots were trained to use AOP strategic resolution guidance, tactical
resolution guidance, and (in the case of manual maneuvers) conflict
prevention information as appropriate to the situation. They were also
instructed to operate the aircraft as they would during line operations.
Although hand-flying was not available, pilots were allowed to use any
desired autopilot modes, including both FMS-coupled modes and tactical
modes.

Results & Discussion

The NASA Langley subject pilots encountered a total of 500 traffic
conflicts throughout the 12 AFR scenarios (C2, C3, and C4). For 332 of
these conflicts, the subject pilot performed a resolution maneuver. The
analyses presented below show results for these conflicts, without
distinguishing between traffic conditions. The effects of traffic
density on resolution performance are treated in a separate publication
(Doble, Barhydt, & Hitt, 2005).

AOP Compliance

To examine the effects of AOP resolution maneuver compliance on
resolution performance, resolution maneuvers were divided into six
categories, based upon whether the maneuver was strategic or tactical
and whether or not the pilot followed AOP guidance. These categories are
summarized in Table 2. Two different performance metrics were then used
to evaluate the maneuvers: induced conflicts and conflicts requiring
multiple resolution maneuvers.

Table 2. Resolution Compliance Categories

  -------------- ---------------------------------------------- -----------
  **Category**   **Description**                                **Count**

  Strategic      Pilot implements AOP-recommended route         141
  Comply         modification without modifications             

  Strategic      Pilot edits waypoints before implementing      0
  Noncomply      AOP-recommended route modification             

  Strategic      Pilot ignores or does not seek AOP resolution, 15
  Manual         and manually edits waypoints, altitudes, etc.  
                 of FMS active route                            

  Tactical       Pilot maneuvers in direction of                118
  Comply         AOP-recommended heading or vertical speed      

  Tactical       Pilot maneuvers away from AOP-recommended      15
  Noncomply      heading or vertical speed                      

  Tactical       AOP tactical guidance not available, pilot     43
  Manual         implements own lateral or vertical maneuver    
                 via autopilot mode control panel               
  -------------- ---------------------------------------------- -----------

*Induced Conflicts*. The frequency of induced conflicts is a measure of
the ability of pilots and AOP to account for aircraft other than the
intruder when calculating a resolution maneuver. An induced conflict was
defined as a new conflict arising within one minute of a previous
resolution maneuver and directly caused by that maneuver. Figure 3 shows
the percentage of resolutions inducing a conflict in each of the six
compliance categories. Results from χ^2^ tests indicate no significant
differences in the frequency of induced conflicts across the three
tactical categories (χ^2^(2, *N* = 176) = 0.27, *p* \> 0.05), but a
significantly higher frequency of induced conflicts for Strategic Manual
maneuvers vs. Strategic Comply maneuvers (χ^2^(1, *N* = 156) = 32.2, *p*
\< 0.05).

![](media/image3.wmf){width="2.9625in" height="1.4708333333333334in"}

Figure 3. Induced Conflicts vs. AOP Compliance

The lowest induced conflict rates occurred when pilots followed AOP
guidance. This highlights the advantage of decision support when
resolving conflicts involving multiple proximate aircraft. It is
conjectured that the relatively high rate of induced conflicts among
tactical maneuvers was due primarily to two factors: the time to
predicted LOS when the maneuvers were executed, and the characteristics
of the AOP tactical resolution algorithm. During the experiment,
tactical resolution maneuvers were generally initiated closer to
predicted LOS than strategic maneuvers. In such situations, especially
in the high-density airspace simulated in this experiment, some induced
conflicts may be inherently unavoidable, as the first priority is
usually to resolve the most critical conflict in a timely manner. In
addition, for very near-term conflicts (under 2 minutes to LOS), the AOP
tactical resolution algorithm did not take other aircraft into account
when calculating resolution guidance. This algorithm was chosen for its
ability to successfully resolve complicated conflict situations without
the need for maneuver coordination between aircraft (Eby, 1994). Ongoing
research will investigate the integration of this algorithm with the AOP
conflict prevention tools in order to further reduce induced conflicts.

[]{.mark}

While the significant increase in induced conflicts for Strategic Manual
resolutions is cause for concern, it should be noted that three of these
five induced conflicts were caused by the same pilot during the same
scenario. Nevertheless, pilot training and the AOP conflict prevention
symbology may warrant further attention as these subject pilots all
implemented route modifications despite being shown Provisional Conflict
Alerts.

*Multiple Resolutions*. The frequency of multiple resolutions is a
measure of the ability of pilots and AOP to resolve a conflict and
remain out of conflict. If a subject pilot was in conflict with the same
intruder multiple times and implemented more than one resolution
maneuver, this was noted as a multiple resolution conflict. Figure 4
shows the percentage of conflicts requiring multiple resolutions in each
compliance category. Results from χ^2^ tests indicate no significant
differences in the frequency of multiple resolutions across the
strategic categories (χ^2^(1, *N* = 156) = 1.67, *p* \> 0.05). The
differences among tactical categories were significant (χ^2^(2, *N* =
176) = 6.04, *p* \< 0.05).

![](media/image4.wmf){width="3.1in" height="1.5131944444444445in"}

Figure 4. Multiple Resolutions vs. AOP Compliance

The lower multiple resolution rate for maneuvers that complied with AOP
guidance (vs. manual maneuvers) shows the benefits of decision support
when resolving conflicts between aircraft flying complex,
four-dimensional trajectories. While the lowest multiple resolution rate
occurred when pilots did not follow AOP guidance (Tactical Noncomply),
this is not seen as a cause for concern. Compliance only accounted for
3% of the variance in multiple resolutions, and this category of
maneuvers had a relatively small sample size. In addition, there may
have been a performance tradeoff, with these maneuvers effectively
avoiding the intruder aircraft at the expense of additional induced
conflicts.

Choice of Maneuver Axis

To judge the relative effectiveness of lateral and vertical AOP
guidance, the maneuvers categorized above as Strategic Comply and
Tactical Comply were further separated into Strategic Lateral, Strategic
Vertical, Tactical Lateral, and Tactical Vertical categories.

*Induced Conflicts. Figure 5* shows the percentage of induced conflicts
that occurred for each of the four axis categories. Results from χ^2^
tests indicate no significant differences between either the strategic
categories (χ^2^(1, *N* = 141) = 0.46, *p* \> 0.05) or the tactical
categories (χ^2^(1, *N* = 118) = 0.37, *p* \> 0.05). For the reasons
mentioned above, it is not surprising that strategic resolutions
resulted in fewer induced conflicts than tactical resolutions, but
within the strategic and tactical categories, the choice of maneuver
axis appears to have had little effect.

![](media/image5.wmf){width="2.80625in" height="1.5458333333333334in"}

Figure 5. Induced Conflicts vs. Maneuver Axis (When AOP Complied With)

*Multiple Resolutions. Figure 6* shows the percentage of multiple
resolutions that occurred for each of the four maneuver axis categories.
Results from χ^2^ tests indicate no significant differences between
either the two strategic categories (χ^2^(1, *N* = 141) = 1.24, *p* \>
0.05) or the two tactical categories (χ^2^(1, *N* = 118) = 0.02, *p* \>
0.05). This shows that lateral and vertical maneuvers were similarly
effective in preventing multiple resolutions. However, the slightly
higher incidence of multiple resolutions for Strategic Vertical
maneuvers is worth noting. These maneuvers required pilots to adjust the
autopilot altitude value in addition to uploading an FMS route
modification. There were cases when the altitude value was not properly
adjusted and the aircraft failed to follow the resolution maneuver.
Compounding this was the difficulty of displaying vertical path changes
on a horizontal situation display. Ongoing research will investigate
other options for presenting vertical maneuver information, including
the use of vertical situation displays.

![](media/image6.wmf){width="2.7875in" height="1.5784722222222223in"}

Figure 6. Multiple Resolutions vs. Maneuver Axis (When AOP Complied
With)

Acceptability of AOP Resolution Guidance

To examine factors that affect pilot perception of the acceptability of
AOP resolution guidance, subject pilots were asked after each scenario
to rate, on a 1 to 7 scale, the acceptability of a) the first AOP
strategic resolution in the scenario, and b) AOP tactical resolutions in
general during the scenario. A series of correlations was then performed
to determine if relationships existed between resolution acceptability
and four other factors: conflict duration, maneuver axis (lateral or
vertical), multiple resolutions, and induced conflicts. These results
are presented in Table 3. Overall resolution acceptability was high for
both strategic resolutions (*M* = 6.31, *SD* = 1.28) and tactical
resolutions (*M* = 5.12, *SD* = 1.60).

Table 3. Resolution Acceptability

+---------------+-------------------+-------------------+-------------+
| **Attribute** | **Acceptability   |                   | **Test**    |
|               | Correlation**     |                   |             |
+---------------+-------------------+-------------------+-------------+
|               | **Strategic       | **Tactical        |             |
|               | Resolutions (*N*  | Resolutions**     |             |
|               | = 109)**          |                   |             |
|               |                   | **(*N* = 135)**   |             |
+---------------+-------------------+-------------------+-------------+
| Conflict      | -0.24\*           | 0.02              | Pearson     |
|               |                   |                   |             |
| Duration      |                   |                   |             |
+---------------+-------------------+-------------------+-------------+
| Maneuver      | -0.18             | 0.10              | Poi         |
|               |                   |                   | nt-biserial |
| Axis          |                   |                   |             |
+---------------+-------------------+-------------------+-------------+
| Multiple      | -0.39\*           | 0.14              | Poi         |
|               |                   |                   | nt-biserial |
| Resolution    |                   |                   |             |
+---------------+-------------------+-------------------+-------------+
| Induced       | -0.02             | -0.17\*           | Poi         |
|               |                   |                   | nt-biserial |
| Conflict      |                   |                   |             |
+---------------+-------------------+-------------------+-------------+
| \* =          |                   |                   |             |
| significant   |                   |                   |             |
| correlation   |                   |                   |             |
| at *p* \<     |                   |                   |             |
| 0.05 level    |                   |                   |             |
+---------------+-------------------+-------------------+-------------+

The acceptability of AOP strategic resolution maneuvers was
significantly correlated with conflict duration and multiple
resolutions. The significance of conflict duration agrees with comments
provided during debrief sessions, which indicated that pilots were
frustrated by AOP computation delays and the options available when AOP
was unable to calculate a solution. While the AOP strategic resolution
algorithm (a genetic algorithm) normally converged on a solution within
one second, insufficient feedback may have been provided to pilots when
computation times were longer, creating the appearance that AOP had
"frozen up." The significant correlation with multiple resolutions is
also reasonable, as one of the primary benefits of intent-based,
strategic decision support is that the necessity for multiple resolution
maneuvers should be reduced by accounting for trajectory changes that
would be unknown to a solely state-based system.

The acceptability of AOP tactical resolutions was only significantly
correlated with whether or not the resolution induced a conflict. As
mentioned above, depending on the time to predicted LOS, the AOP
tactical guidance may or may not have accounted for aircraft other than
the intruder. As such, there were cases when the tactical guidance
disagreed with Maneuver Restriction Bands. Although this behavior was
explained to subject pilots during training exercises, this is
recognized as a significant human factors issue. Research is underway to
modify the AOP near-term tactical resolution logic so that conflicting
information is not presented to pilots.

Practice Effects

The En Route Free Maneuvering experiment lasted a total of eight days,
with three days devoted to training, four days for data collection, and
one day for debriefing. Each data collection day included four
scenarios, with one at each traffic condition, and with the order of
conditions varying across days.

To identify any learning or practice effects, conflicts were sorted by
day and evaluated with the same performance metrics presented above.
Figure 7 shows the frequency of induced conflicts and multiple
resolutions across days. χ^2^ tests indicate that no significant
differences in the frequency of induced conflicts (χ^2^(3, *N* = 332) =
1.37, *p* \> 0.05) or in the frequency of multiple resolutions (χ^2^(3,
*N* = 332) = 4.78, p \> 0.05) existed across the four days.

![](media/image7.wmf){width="2.910416666666667in"
height="1.6111111111111112in"}

Figure 7. Resolution Performance by Day

While no significant practice effects were found, it is interesting to
compare the performance by day with the resolution maneuvers chosen.
Figure 8 shows the percentage of maneuver types chosen each day.
Notionally, resolution performance appears to degrade with increases in
manual and non-complying maneuvers over the first three days of the
experiment, then improve on Day 4 with an increase in Strategic Comply
maneuvers.

![](media/image8.wmf){width="3.0145833333333334in"
height="1.5263888888888888in"}

Figure 8. Resolution Maneuver Compliance by Day

Conclusions

Through the above analysis of conflict resolution maneuvers, several
conclusions can be drawn about the performance of pilots during
AOP-equipped AFR operations. First, the choice of maneuver axis (lateral
or vertical) had little effect on resolution performance, indicating
that resolution maneuvers can be well-executed in either axis. Second,
resolution performance was shown to generally improve when pilots
complied with AOP-recommended resolution maneuvers. Finally, although
pilot acceptability of AOP guidance was high overall, possible ways to
further increase acceptability and performance were identified. These
methods include better integration of AOP near-term tactical resolution
logic with conflict prevention information, improved feedback when AOP
cannot converge on a strategic solution, and the potential inclusion of
a vertical situation display. Along with previous findings, these
results further support the feasibility of the En Route Free Maneuvering
concept while highlighting areas for future research.

Acknowledgements

The authors would like to thank James Hitt at Booz Allen Hamilton for
performing statistical analyses of the experimental results, and the
subject pilots and controllers for their participation in the
experiment.

References

Barhydt, R., & Kopardekar, P. (2005). *Joint NASA Ames/Langley
Experimental Evaluation of Integrated Air/Ground Operations for En Route
Free Maneuvering*. Submitted to 6^th^ USA/Europe Air Traffic Management
R&D Seminar, Baltimore, MD.

Barhydt, R., & Krishnamurthy, K. (2004). *Design of a Multi-Mode Flight
Deck Decision Support System for Airborne Conflict Management.* Intl.
Conference on Human-Computer Interaction in Aeronautics (HCI-AERO),
Toulouse, France.

Doble, N.A., Barhydt, R., & Hitt, J.M. (2005). *Conflict Resolution
Performance in an Experimental Study of En Route Free Maneuvering
Operations.* Submitted to 6^th^ USA/Europe Air Traffic Management R&D
Seminar, Baltimore, MD.

DOT Joint Planning and Development Office. (2004). *Next Generation Air
Transportation System Integrated Plan*. Washington, DC.

Eby, M.S. (1994). *A Self-Organizational Approach for Resolving Air
Traffic Conflicts.* The Lincoln Laboratory Journal, Vol. 7, Nr. 2.

FAA. (2004). *Forecast Fact Sheet Fiscal Years 2004-2015*. Washington,
DC. [http://www.faa.gov]{.underline}

[/newsroom/factsheets/2004/factsheets_040325.pdf]{.underline}

Hoekstra, J.M., et al. (2000). *Overview of NLR Free Flight Project
1997-1999*. Contract Report NLR-CR-2000-227. National Aerospace
Laboratory, Amsterdam, The Netherlands.

Krishnamurthy, K., et al. (2003). *Autonomous Aircraft Operations Using
RTCA Guidelines for Airborne Conflict Management.* 22^nd^ Digital
Avionics Systems Conference, Indianapolis, IN.

NASA Advanced Air Transportation Technologies Project Office. (1999).
*Concept Definition for Distributed Air/Ground Traffic Management
(DAG-TM) Version 1.0.* NASA Ames, Moffett Field, CA.