![](media/image1.wmf)

U.S. Department

of Transportation

Research and Innovative Technology

Administration

Volpe National Transportation

Systems Center

Report of a Workshop on

Human-Automation Interaction in NGATS

![](media/image2.jpeg){width="0.6590277777777778in"
height="0.5618055555555556in"}

NASA Airspace Systems Program

Final Report

October 2006

Thomas B. Sheridan, Kevin M. Corker and Eric D. Nadler

+-----------------------------------------------------------------------+
| > Notice                                                              |
|                                                                       |
| This document is disseminated under the sponsorship of the Department |
| of Transportation in the interest of information exchange. The United |
| States Government assumes no liability for its contents or use        |
| thereof.                                                              |
+-----------------------------------------------------------------------+
|                                                                       |
+-----------------------------------------------------------------------+
| > Notice                                                              |
|                                                                       |
| The United States Government does not endorse products or             |
| manufacturers. Trade or manufacturers' names appear herein solely     |
| because they are considered essential to the objective of this        |
| report.                                                               |
+-----------------------------------------------------------------------+

![](media/image1.wmf)

![](media/image1.wmf)

TABLE OF CONTENTS

[**1.0 INTRODUCTION AND PURPOSE OF THE WORKSHOP
1**](#introduction-and-purpose-of-the-workshop)

[2.0 BRIEF OVERVIEW OF NGATS 3](#brief-overview-of-ngats)

[3.0 HUMAN-AUTOMATION INTERACTION ISSUES
5](#human-automation-interaction-issues)

[3.1 Air-Ground Human-Automation Collaborative Decision-Making in NGATS
5](#air-ground-human-automation-collaborative-decision-making-in-ngats)

[3.1.1 Air-ground Human-Automation Issues and Research Needs by Phase of
Flight
6](#air-ground-human-automation-issues-and-research-needs-by-phase-of-flight)

[3.1.2 Generic and System Design Issues
8](#generic-and-system-design-issues)

[3.2 Net-Centric Information Access Use and Management in NGATS
10](#net-centric-information-access-use-and-management-in-ngats)

[3.2.1 What information should be "pushed" and what information should
be "pulled"?
10](#what-information-should-be-pushed-and-what-information-should-be-pulled)

[3.2.2 How should net-centric information be "tagged"?
11](#how-should-net-centric-information-be-tagged)

[3.2.3 Issues relating to use of voice vs. datalink
11](#issues-relating-to-use-of-voice-vs.-datalink)

[3.3 Selection and Training of Air Traffic Managers for NGATS and
Training for New Flight Deck Operations
12](#selection-and-training-of-air-traffic-managers-for-ngats-and-training-for-new-flight-deck-operations)

[3.3.1 Technical background 12](#technical-background)

[3.3.2 New or different training methods for automation supervisors
12](#new-or-different-training-methods-for-automation-supervisors)

[3.3.3 Different training for controllers staffing different domains of
airspace
12](#different-training-for-controllers-staffing-different-domains-of-airspace)

[3.3.4 Crew resource management extended from the flight desk to the
controller community and the aircrew
12](#crew-resource-management-extended-from-the-flight-desk-to-the-controller-community-and-the-aircrew)

[3.4 Planning the Transition from the Current National Airspace System
to NGATS (2020-2025)
13](#planning-the-transition-from-the-current-national-airspace-system-to-ngats-2020-2025)

[3.4.1 Modularizing NGATS subsystem development and certification in
bite-size increments.
13](#modularizing-ngats-subsystem-development-and-certification-in-bite-size-increments.)

> [3.4.2 Organizational/cultural impediments to be overcome in the
> transition to NGATS
> 13](#organizationalcultural-impediments-to-be-overcome-in-the-transition-to-ngats)

[3.5 Research Methodologies Appropriate for NGATS Human-Automation
Interaction
13](#research-methodologies-appropriate-for-ngats-human-automation-interaction)

[3.5.1 Task analysis 14](#task-analysis)

[3.5.2 Lessons learned from research and implementation of
human-automation systems in previous aviation and other contexts
14](#lessons-learned-from-research-and-implementation-of-human-automation-systems-in-previous-aviation-and-other-contexts)

[3.5.3 Human-in-the-loop and fast-time simulations
14](#human-in-the-loop-and-fast-time-simulations)

[3.5.4 Risk analysis 16](#risk-analysis)

[4.0 PRIORITY RESEARCH NEEDS 17](#priority-research-needs)

[4.1 Performance of Humans Interacting with Automation and Optimization
Processes
17](#performance-of-humans-interacting-with-automation-and-optimization-processes)

[4.2 Single and Mixed-Mode Communication and Negotiation
17](#single-and-mixed-mode-communication-and-negotiation)

[4.3 Individual and Team/Group Situation Awareness Issues
17](#individual-and-teamgroup-situation-awareness-issues)

[APPENDIX A. QUESTIONS POSED TO PARTICIPANTS
19](#appendix-a.-questions-posed-to-participants)

[APPENDIX B. ATTENDEES 21](#appendix-b.-attendees)

[5.0 ACKNOWLEDGMENTS 23](#acknowledgments)

# INTRODUCTION AND PURPOSE OF THE WORKSHOP

A workshop was held on May 10 and 11, 2006, at the conference facilities
of the Georgia Institute of Technology Research Institute in Arlington,
Virginia, under the sponsorship of the Next Generation Air
Transportation System (NGATS) Joint Planning and Development Office
(JPDO). The purpose of the workshop was to bring together NASA, FAA,
JPDO and academic engineers and scientists to discuss what research is
needed in the coming few years in order to understand the
human-automation interactions posed by NGATS concepts of operation
sufficient for system design. The workshop was organized as part of a
NASA funding agreement with the John A. Volpe National Transportation
Systems Center to identify human-automation issues and research needs in
NGATS.

While NGATS will involve much greater automation compared to the current
air transportation system, human pilots, air traffic service providers,
and other system operators will still be required to monitor operations
and ensure safety, especially in the case of hardware or software
failures or other unanticipated events. These human operators will have
to assume new roles and responsibilities. Thus there are many issues of
human-automation interaction that have not been sufficiently researched
and remain unresolved with respect to design requirements. This includes
security issues that are intentionally not raised in this report.

The workshop began with presentations by R. D. Arbuckle, head of the
JPDO Integrated Project Team responsible for Agile Air Traffic Control,
and J. A. Cavolowsky, Deputy Program Manager, NASA Airspace Systems
Program. The remainder of the meeting was devoted to discussion,
organized around five topics:

-   Air-ground human-automation collaborative decision-making in NGATS

-   Net-centric information access, use, and management in NGATS

-   Selection and training of air traffic managers for NGATS

-   Planning the transition from the current national airspace system to
    NGATS (2020-2025)

-   Research methodologies appropriate for NGATS human-automation
    interaction

Following each discussion topic, participants were asked to write down
what they regarded as the most salient issues and research needs. These
comments were collected and organized. This report is largely based on
those comments as well as the points made during free discussion. The
issues are expressed here mostly as questions concerning
human-automation interaction yet to be resolved and probably in need of
research for their resolution. We emphasize that the intent at this
point is to faithfully summarize the meeting discussion and not to draw
conclusions for a research program.

Appendix A lists the detailed questions asked of the participants prior
to the workshop. Appendix B lists the participants, their organizations,
and email addresses.

# BRIEF OVERVIEW OF NGATS

In 2003, President Bush and Congress took a critical step towards
transformation of the air transportation system with the enactment of
VISION 100 -- Century of Aviation Reauthorization Act, which laid out a
mandate for the multi-government-agency planning project called Next
Generation Air Transportation System (NGATS), and formally created the
Joint Planning and Development Office (JPDO) to manage the work
associated with it. The overarching vision was for a system that
addresses critical safety and economic needs in civil aviation for
2020-2025, such as increased capacity, while fully integrating
environmental impact, national defense, and homeland security
improvements -- and in a cost effective manner.

NASA has strategic goals that relate to the work of the JPDO to develop
NGATS. NASA's overall mission is to: "Pioneer the future in space
exploration, scientific discovery, and aeronautics research." The NASA
Aeronautics Research Mission Directorate is responsible for achieving
NASA Strategic Goal 3E, "Advance knowledge in the fundamental
disciplines of aeronautics and develop safer aircraft and higher
capacity airspace systems." In particular, NASA's Airspace Systems
Program has the goal of conducting cutting-edge research needed for the
development of the high capacity, efficient, and safe airspace and
airportal systems that will comprise NGATS, in partnership with the
JPDO.

The JPDO has identified eight "key capabilities" which will characterize
NGATS:

-   Network Enabled Information Access -- This capability will make
    information available, securable, and usable in real-time according
    to defined "communities of interest."

-   Performance-Based Services -- NGATS will provide multiple levels of
    service (i.e., service tiers) where operators can select the
    performance-based service level they desire as long as they can meet
    equipage, aircraft performance, training, and security requirements
    needed for that level of service. The services will vary from area
    to area and will vary with time as needs dictate.

-   Weather Assimilated into Decision Making -- There will be an
    enhanced common weather picture across all service providers and
    users that will be assimilated into NGATS decisions.

-   Layered, Adaptive Security -- NGATS will provide embedded and
    interwoven security layers that operate seamlessly and adapt to
    changing situations.

-   Broad-Area Precise Navigation -- Satellite-based precision
    navigation service will be provided where and when needed, and in
    particular, instrument landings will be possible at any airportal.

-   Aircraft Trajectory-Based Operations -- The future system will use
    four-dimensional (4D) gate-to-gate trajectories as the basis for
    planning and executing system operations. This capability also
    contains an "Evaluator," which is an integrated computer-based
    system to support planning and executing Aircraft Trajectory-Based
    Operations.

-   Equivalent-Visual Operations -- Aircraft will perform "equivalent
    visual" operations in non-visual conditions. Availability of the
    Equivalent Visual Operations capability at all airportals means more
    airports can provide reliable air service.

-   Super Density Operations -- NGATS will provide peak performance for
    the busiest airports. There will be reduced arrival/departure
    spacing, equivalent visual capability, reduced runway occupancy
    time, simultaneous operations on single runway, and integration of
    better tools to detect and avoid wake vortices.

Thus the vision is for a system that accommodates many more air
travelers. Much more accurate surveillance would be achieved by means of
Automatic Dependent Surveillance-Broadcast (ADS-B) or successor
satellite positioning and communication in lieu of radar. High altitude
airspace would be restricted to aircraft specially equipped, and control
of these aircraft, both in the air and on the airport surface, would be
automated to a much greater degree than today. Pilots of these aircraft
would assume a greater role for self-separation. Providers of air
traffic services for these aircraft would assume a more supervisory role
rather than vectoring individual aircraft. Air-ground communication
would be mostly via digital datalink rather than voice. Aircraft that
are unequipped would be accommodated in mostly the currently
conventional way by air traffic management, though restricted with
respect to allowable airspace.

For more detail on the JPDO and its developments please see:

http://www.jpdo.aero/site_content/index.html.

# HUMAN-AUTOMATION INTERACTION ISSUES

This section summarizes the human-automation interaction issues
identified and discussed by the workshop participants. It is organized
by the five topics listed in Section 1.0 that were discussed at the
workshop.

At various places in the listing below the editors of this report refer
in italics to examples of corresponding NASA research milestones for the
FY07-11 planning horizon in the Air Traffic Management - AirSpace (AS)
project[^1] and the Integrated Intelligent Flight Deck (IIFD)
project[^2]. (At the time of this writing the Air Traffic Management -
Airportal project milestones have not been issued).

We note that at the most fundamental research level (indicated with the
number 1 as the first digit of the milestone designations below) "human
factors" is a major category in the AS project and "operator
characterization" is explicitly named as a major category in the IIFD
project. At the more applied levels 2 and 3, though human-automation
related issues are not explicitly spelled out to any extent,
human-automation concerns cut across many or even most of the major
research categories.

Because of the phase-of-flight organization used in the workshop, the
NASA milestones do not correlate one-to-one with the workshop topics. No
claim is made that all relevant NASA milestones are cited, and several
may be cited more than once as related to the research needs identified
in the workshop.

## Air-Ground Human-Automation Collaborative Decision-Making in NGATS

Automation is expected to play a critical role in the proposed
achievement of airspace transportation improvement. Use of computers is
envisioned to help negotiate user preferences, support human
decision-making, respond to anomalies, and balance constraints and
demand throughout the NGATS operations. While the roles and
responsibilities of humans and computers have yet to be defined, the
NGATS operational concepts assert that humans and automation aids will
operate cooperatively to achieve the desired operational effectiveness
and safety. We have organized this particular discussion of
human-automation collaborative decision-making by phase of flight in
which that automation plays a role and by cross-cutting issues.

### Air-ground Human-Automation Issues and Research Needs by Phase of Flight

There are four phases of flight during which human-automation
interaction is seen as being most critical: (1) Preflight trajectory
negotiations; (2) Airportal surface operations; (3) Airspace trajectory
negotiations for climb-out, transition between segments of airspace, and
descent; and (4) 4D trajectory modifications due to weather or other
abnormal circumstances. Many air-ground human-automation collaborative
decision-making issues are unique to a particular one of these four
phases of flight:

#### Preflight trajectory negotiations

a)  Preflight negotiations will be accomplished at several stages. From
    one to many weeks prior to a flight the airline operations centers
    (AOCs) will negotiate with the air traffic service provider to reach
    agreement on a 4D enroute trajectory. The negotiation will be
    supported by a shared use of the Evaluator. The 4D en route
    trajectory will be updated at least once, days to minutes preflight,
    to (possibly) revise the plan. The question is how will this
    negotiation take place: how will a plan be presented along with
    reasons why that plan was chosen, and what should the procedure be
    to negotiate the plan?

b)  Presumably computer systems will somehow participate in the
    negotiation between airlines, pilots and air traffic management
    (ATM), but what limitations can the computer impose on flexibility,
    e.g., time allowed to come to agreement or extent of the deviation
    from the optimal system solution. What happens if the AOC/pilot
    rejects the computer proposal, for example how many modification
    attempts are acceptable? Under what circumstances, if any, should a
    human representing ATM step in? Can negotiations include monetary or
    other trades (e.g., of slots)?

c)  How are automation failures or partial failures identified? What is
    the backup to automation failures?

> *No milestone deals specifically with preflight negotiations, nor is
> the differing nature of negotiations at different phases of flight
> (preflight, on the surface, climbout and descent, and enroute)
> detailed in the milestones. Many milestones relate to "Evaluator
> development", which necessarily includes user negotiation
> considerations. AS3.4.01-06 includes "determining appropriate roles
> and procedures that enable users and air traffic service providers" to
> operate the system, including balancing workload. IIFD1.6.04 is to
> assess "hazard detection and severity estimation."*

#### Airportal surface operations

a)  What should be the assignments in terms of roles and
    responsibilities for ramp, ground, and local controllers vis-à-vis
    the surface computer planner? Should they communicate with the
    aircrew and AOC via datalink, voice, or mixed-mode? How does the
    computer keep track of voice communication? Are the operators
    required to enter data on decisions? How will handoffs be handled?

b)  How will the computer convey the taxi plan to the aircrew
    (datalink/voice, spatial displays like moving maps, head-up
    displays, or some combination)? Does the aircrew have any options to
    modify the plan?

c)  To what extent will the taxi operation be controlled manually by the
    pilot, and to what extent automatically through the Flight
    Management System (FMS) (presumably with the pilot monitoring and
    able to take over control)? How will information about movement of
    other aircraft, other vehicles, wake vortices, or errors in movement
    of own aircraft be communicated to aircrew and/or human ATM
    personnel? Who has responsibility for surface separation?

d)  How will surface actions be controlled: holds at ramps, on taxiways,
    runway crossings, or on runways; takeoff clearances; takeoffs? What
    role will the computer play? What role will human controllers play?
    Will the human controller or computer intervene to prevent impeding
    collisions?

e)  What decision support tools are appropriate for pilot or controller
    decisions in surface operations?

f)  What is the backup to automation failures in any of the above?

> *Airportal milestones are not yet available at this writing.
> Milestones AS3.6.01-06 concern airportal operations, including human
> cognition and workload.*

#### Airspace trajectory negotiations for climb-out, transition between segments of airspace, and descent

a)  Assignment and expectation of responsibilities for separation is
    critical. Supposedly the pilot will self-separate and/or perform
    station-keeping maneuvers, but what are his or her degrees of
    freedom in doing so? How is the pilot informed that deviation from
    the accepted trajectory is too great, especially if in conflict with
    another aircraft? How far away from the agreed 4D trajectory may the
    pilot deviate without being contacted by the air navigation service
    provider, alerted by on-board equipment, or declaring an emergency?
    Under what circumstances should the controller step in, and by what
    procedure and communication mechanism does he or she do so?

b)  What decision support tools are appropriate for pilot or controller
    decisions in climb-out, transitions within airspace, or descent
    operations? For example how should the pilot or controller be helped
    to anticipate dynamic changes in airspace configuration and
    understand the reasons for it?

c)  How are equity issues between aircraft (that affect time, fuel, and
    turbulence considerations) to be resolved? Which humans should be
    participants in the decisions (pilot, sector controller, AOC, flow
    control, tactical computer, Evaluator)?

d)  What is the backup to automation failures in any of the above?

> *Milestone AS1.4.04 seeks a metric for traffic complexity as it
> affects controller workload. AS1.6.01-04 concern safety of descent
> operations and arrival scheduling in the superdensity environment.
> AS2.4.01-11 relates to information management to ensure separation as
> well as meet user needs. Human-automation allocation is specifically
> included. IIFD1.4.02 and IIFD1.4.06 are about display formats to meet
> information needs, essential to in-flight negotiations and
> decision-making. IIFD3.3.01 concerns operator attention management in
> approach and landing.*

#### 4D trajectory modifications due to weather, sector saturation, or other abnormal circumstance

a)  Which decision support tools are appropriate for pilot or controller
    to anticipate weather conditions that presage trajectory changes?

b)  How are trajectory changes for weather to be negotiated? What will
    be the roles for the pilot, sector controller, flow control, AOC,
    tactical computer controller, and Evaluator? Who has final
    authority? Will communication be via voice, datalink, or a mixture?

c)  How are equity issues (that affect time, fuel, turbulence
    considerations) in rerouting to be resolved? Which humans should be
    participants in the decisions (pilot, sector controller, AOC, flow
    control, tactical computer, Evaluator)?

d)  What is the backup to automation failures in any of the above?

> *Milestones AS1.7.01-06 deal with responsibilities between human and
> automation separation assurance, and the potential for pilots to game
> the system to their own advantage. AS 2.4.04 seeks research on
> human-automation function allocation related to the Evaluator.
> AS2.5.01-03 are aimed at human-automation functional analysis and
> getting input from service providers on operational concepts to
> improve separation assurance . IIFD3.3.02-07 are for sensing and
> display of hazard information to improve operator situation
> awareness.*

### Generic and System Design Issues

There are generic human behavior issues and generic system design issues
that cut across all of the previous four phases of flight groupings:

#### Generic human behavior issues

a)  Understanding what the automation is supposed to do or can be
    expected to do, what it is doing now, or probably will do next.

> *There appears to be little emphasis on "understanding the automation"
> in the research milestones, though that can be considered part of
> situation awareness as covered by several of the milestones cited
> above. Perhaps that issue is relegated to FAA pilot and controller
> training. IIFD1.3.01 deals with understanding operational hazards.
> IIFD1.5.04 concerns crew training protocols.*

b)  Remembering task demands and salient information required for future
    task actions in spite of intervening task demands (i.e., where
    stimuli and appropriate responses for a task are interleaved with
    stimuli and appropriate responses for other tasks). This is called
    prospective memory, and is a serious problem in multi-tasking.

> *Prospective memory is related to attention, situation awareness,
> workload, complexity and other factors named in the milestones* as
> covered, for example, by *AS2.6.07, particularly with regard to
> superdensity operations.* *IIFD1.5.11 is explicitly concerned with
> task demands.*

c)  Monitoring and maintaining situation awareness over long and boring
    periods of nominal operations under automatic control (with a
    possible need to impose activities for the purpose of maintaining
    alertness). Some believethat pilots and controllers will be no more
    busy than they are today, or that a high degree of automatic control
    will not occur.

d)  Ability during very busy times, especially in anomalies involving
    multiple aircraft, to attend to multiple inputs and maintain
    situation awareness, and avoid breakdown due to excessive cognitive
    workload.

e)  Transitioning from low to high cognitive load when circumstances
    demand it.

f)  Knowing who has responsibility for overall system safety.

g)  Knowing the operating state of the system and the rules and
    operational concepts that are controlling the current actions?

> *Issues c through g correlate with milestones IIFD1.1.11 and
> IIFD1.1.15, which deal with measuring operator state, and by inference
> an operator's ability to know system state, especially to recognize
> hazardous states. IIFD1.8.04 deals with operator monitoring
> requirements. IIFD2.2.06b relates to displaying information
> uncertainty. IIFD2.2.10 specifically seeks to model situation
> awareness. IIFD2.4.03- 09 are concerned with sensing and monitoring
> operator state.*

#### System design issues

There are also generic system design issues:

a)  How to design for scalability, i.e. the robustness of the system to
    handle fewer or greater numbers of aircraft, smaller or larger
    aircraft, fewer or greater numbers of airports and runways, etc?
    What are appropriate constraints and benchmarks on scalability from
    a human factors perspective, i.e., how will attention and workload,
    situation awareness, and memory be affected in scaling?

> *Scalability is a problem of managing system design and redesign over
> time. It does not appear to be emphasized in current NASA milestone
> definition.*

b)  If continuous computer-based decision-making substitutes for human
    controllers who in the past have been positioned in "outer" control
    loops to control aircraft trajectory, there is the possibility of
    time delay or intermittent sampling because of controller
    multi-tasking. (The "inner" loop is the pilot/autopilot controlling
    the aircraft in pitch, roll, yaw and airspeed). This poses a danger
    of dynamic instability. Delayed outer loop decisions might occur,
    for example, when sectors become overloaded, or airspace is
    reconfigured .

> *Instability is a category of hazard. Milestone AS3.3.03 seeks to
> evaluate the level of traffic that can be safely handled in a sector.
> AS3.3.04 is concerned with the frequency of airspace reconfiguration
> and its effect on operators.*

c)  How are metrics of operational complexity relevant and how are they
    used, e.g., metrics like number of modes of display or control,
    number of procedural contingencies, or number and rate of airspace
    or operating configuration changes? How well can these metrics
    predict system safety and efficiency?

> *Complexity occurs at different scales. Milestone AS1.4.04 and
> AS2.4.02 seek to define and cope with traffic and airspace complexity.
> AS1.04.05 and IIFD2.2.06b are about information uncertainty.
> IIFD2.2.12 relates to display clutter. AS3.3.03 seeks to understand
> complexity limits for different classes of airspace as related to
> operator workload.*

Probing these issues must be done mostly with simulations in which human
operators interact with each other and/or with the appropriate
technology to perform assigned tasks. Most of the simulations will be
with real and well-trained human subjects, so-called human-in-the-loop
simulations; the simulated environment and automation technology will be
embodied in computers. For some simulation tasks sufficiently accurate
mathematical models of the human operator are available that the whole
exercise can be done in a computer. More will be said of simulation
approaches later in this report.

## Net-Centric Information Access Use and Management in NGATS

A distinguishing feature of the planned NGATS is so-called net-centric
information, which will be broadly available via digital communication
links to humans and computer users in the system. Some of it will be
real-time, and some stored or buffered at various ground or airborne
locations. This includes ADS-B surveillance communication as well as
other datalink communications between aircraft and ground. The US
Department of Defense has pioneered net-centric information systems, but
application to NGATS poses many new questions:

### What information should be "pushed" and what information should be "pulled"?

What information should necessarily be presented to the aircrew and/or
the air traffic controllers, and what information should be available
only by being accessed at their will?

a)  How should aircrew and air traffic managers know what information is
    available on the net?

b)  How will they know whether to expect it to be pushed or whether they
    will have to pull it?

c)  How should they be alerted when they should access information, or
    when frequently or recently used information has been updated?

d)  How will information available to be pulled be organized and
    accessed? Will keywords be used like Google® or some hierarchical
    system like a conventional library, or one based on functional
    contingencies, e.g., the airspace in which the access is being
    undertaken?

e)  How should pushed information be organized? What should be the
    tradeoff between time-criticality and importance? Can "information
    value" decision analysis be fruitfully applied to this problem?

f)  What information should be unavailable to which person or computer?

g)  Should operators in the system be able to regulate levels of detail
    or quantity of information?

h)  On the integrated flight deck, should datalinked information be
    displayed in a common place, or distributed among existing aircraft
    displays?

### How should net-centric information be "tagged"?

a)  To indicate relevance to normal operation or emergency procedure,
    especially with respect to recent changes in the system?

b)  To indicate current estimate of accuracy or reliability?

c)  To indicate trends as well as predicted values, including predicted
    accuracy or reliability?

d)  Should it specify who is expected to act on the information and/or
    who has authority to act?

### Issues relating to use of voice vs. datalink

a)  Under what circumstances should voice be used?

b)  How important is the current "party line" (same voice frequency) of
    shared information for situation awareness, and need this be
    continued in NGATS, or can its advantages be replaced in some form
    on datalink?

c)  How will typing speeds and typing errors of pilots or air traffic
    managers affect datalink communications?

d)  How will visual attention demands of datalink communication on
    pilots or air traffic managers suffer from other visual attention
    demands placed on them?

> *Various milestones concerned with information requirements relate to
> all of the net-centric information issues identified by the workshop.
> IIFD1.4.02 concerns display formats and media to achieve visual/
> flight deck information requirements. IIFD3.2.2 deals with information
> needs, while IIFD3.2.3 is about how to display it. The question of
> what is "pushed", what is "pulled" and what is restricted from some
> persons in the system is not emphasized in the NASA plans.*

## Selection and Training of Air Traffic Managers for NGATS and Training for New Flight Deck Operations

A current wave of retirements of air traffic controllers and managers is
posing serious challenges regarding selection and training of future FAA
air traffic personnel. Selection criteria and training probably should
be modified somewhat, in view of the increased amount of automation and
the changed role of ATM personnel in the NGATS 2020-2025 time frame,
even though that remains 14 years away. Some anticipated issues are:

### Technical background

Will the normal computer and automation experience of the new generation
of prospective controllers adequately prepare them, or must selection
criteria be made more stringent with regard to formal technical
education?

### New or different training methods for automation supervisors

What new or different training methods should be developed for NGATS
controllers and flight crew given that the controller role will change
to monitoring, management by exception, and supervisory control -- and
away from vectoring individual aircraft? On-the-job training may be
inadequate.

a)  How to train, and at what appropriate level, for understanding of
    the automation and failure detection systems?

b)  How to design simulator refresher training with random failures and
    anomalies?

### Different training for controllers staffing different domains of airspace

Should controllers rotate through serving these different domains so
that skills in individual aircraft vectoring are maintained (in case
NGATS automation goes down)? Should the controllers' training be
differently organized into functional categories such as high altitude,
high automation, or transition airspace?

### Crew resource management extended from the flight desk to the controller community and the aircrew

Is enlarging the "team" to include both pilot and controller impractical
due to different organizational loyalties?

> *Milestone AS.2.7.02 is to develop predictive, conceptual-level,
> safety assessment methods for ill-defined complex interacting systems
> including the National Airspace System (NAS). This includes procedural
> training. IIFD 1.5.04 is about validation of operational protocols and
> crew training guidelines for the integrated flight deck.*

## Planning the Transition from the Current National Airspace System to NGATS (2020-2025)

This transition poses many challenges. NGATS cannot possibly come into
being all at once. Development, proof-testing, and FAA approval of
various systems will occur in stages. For economic and system
availability reasons, different air carriers and general aviation
operators will install different NGATS compatible equipment at different
stages in time. Somehow the system will have to adapt to this evolution
over the next two decades. Among the human-automation interaction issues
that occur due to transition are:

### Modularizing NGATS subsystem development and certification in bite-size increments.

a)  How much human-in-the-loop fidelity is necessary at each level of
    development?

b)  Should a suite of human-automation interaction options be maintained
    until much or most of the NGATS development is completed?

### Organizational/cultural impediments to be overcome in the transition to NGATS

a)  How to evolve to a safety culture that encourages reporting of
    errors and inefficiencies, with concurrent vertical collaboration,
    in contrast to the current adversarial arrangement?

b)  What effect will co-location of centers and TRACONs have on system
    operations?

c)  How to maintain human factors community participation during the
    transition to provide timely design requirements to the system
    architects and designers in an understandable language?

> *As suggested earlier, the workshop participants indicated that
> research on management of project design to accommodate transition is
> important, but it is not emphasized in the NASA planning documents.*

## Research Methodologies Appropriate for NGATS Human-Automation Interaction

A principal goal of the workshop was to identify issues that require
research to be done before design requirements can be stated. A part of
specifying research needs is deciding what are the most appropriate
research methods.

> *This is parallel to IIFD statement of purpose: IIFD tracks
> improvements in predictive capability through continuous
> identification and assessment of areas of uncertainty that require
> investigation. This is accomplished through the development of system
> design tools; the application of system analysis and data mining
> techniques that identify phenomena of interest and hazard precursors;
> and the definition of validation metrics that quantify uncertainties
> for system and subsystem performance and integrity (IIFD Plan, p.3).*

### Task analysis

Research on any aspect of human-automation interaction presupposes that
the task of the human is well specified. This implies that a cognitive
task analysis be performed to translate NGATS concepts-of-operation
documents into some level of description of what information the human
pilots and air traffic managers must acquire, what decisions they must
make, and what actions they must take. Such task analysis cannot wait
until system design is in final stages. Therefore the methods for
supporting these task analyses will have to be robust at varied levels
of definition and detail, for the research depends on the cognitive task
analysis and the system design depends on the research. Thus the
cognitive task analysis will have to be iterative.

### Lessons learned from research and implementation of human-automation systems in previous aviation and other contexts

An important step is for researchers to perform literature review and
consideration of past human-automation interaction failures and lessons
learned (e.g., military, nuclear and chemical process control, robotics
and manufacturing systems, hospital and business systems, etc.).

> *Milestone IIFD3.2.2 is to validate flight deck guidelines and
> information and display requirements through assessment of usability,
> acceptability, suitability, and safety of first generation adaptive
> display and interface technologies.*

### Human-in-the-loop and fast-time simulations

Extensive simulations that include human operators, communication and
control automation, and some features on the environmental context to
add realism, must be done. Many or most of these simulations should be
human-in-the-loop with human subjects that are credibly experienced as
pilots or controllers.

> *AS.1.5.04 is to develop methods for quantifying the safety level of
> human operators in Air Traffic Management (ATM) systems, using
> simulation for validation in the presence of uncertainties. AS.3.5.06
> expands human-in-the-loop simulation of service-provider-based
> automated situation awareness to focus on controller/pilot roles and
> responsibilities for time-based metering with automated separation
> assurance, including failure and recovery models, mixed equipage
> operations, and human/machine allocations. AS.4.5.01 is for simulation
> analysis of service-provider-based automated separation assurance with
> complex traffic, metering, hazardous weather, and failure recovery.*

Where computational models of human operators are sufficiently well
developed and robust operator models can be connected to models of the
automation, aircraft, etc., so that fast-time simulation experiments can
be conducted.

> *Milestone AS.2.7.01 is to develop a method for modeling human
> workload in fast-time simulations and validate models against workload
> measurements. IIFD1.5.7 will use validated models of attention
> allocation and prospective memory to develop error mitigation
> strategies. IIFD1.8.1will develop models of distributed
> operator/automation systems, including definition of desired/required
> safety properties. IIFD1.8.4 should provide formal models of operator
> monitoring requirements, including abstractions of operator
> \"failure\" modes. IIFD1.8.6 is to define/refine computational models
> for prediction of human-automation integration vulnerabilities of
> flight deck technologies. IIDF2.2.2 is to develop models to analyze
> the effects of operational conditions, task demands, and
> organizational policies and procedures on vulnerability to pilot
> errors and accidents in the NGATS highly automated environment.*

a)  Because high-fidelity human-in-the-loop simulations can be very
    expensive to set up and run, it is recommended that initial
    simulations be just realistic enough to provide the subjects a good
    idea of what the task is. Much can be learned from such crude
    simulations. Eventually, of course, high-fidelity simulations and
    testing with actual systems must be performed to verify performance
    and safety.

> *Milestone IIFD3.2.2 assesses initial multi-modal presentation formats
> and interaction methods for 4D plus uncertainty display concepts and
> virtual visual environments. IIFD2.4.7 calls for a low fidelity
> simulation study to investigate operator engagement indices for
> different levels of human/ automation integration.*

b)  Before programming computers to implement even crude simulations it
    is sometimes useful to utilize role-playing games. For example in
    the "Wizard of Oz" technique humans play the role of the computer,
    responding to control decisions or information queries by indicating
    system responses or providing requested information.

> *In the IIFD program the Tailored Flexible Operator-Automation
> Management (TFOAM) project element addresses the requirement to
> perform a task and function analysis beyond traditional simulation
> studies. "Formal methods analyses allow for a thorough evaluation of
> allocation concepts beyond traditional simulation studies. The
> research provides a conceptual framework that can accommodate new
> technologies as well as new operator roles as they emerge." Milestone
> IIFD3.1.1 asks for preliminary assessment of functional categories and
> operational scenarios, and cross-referencing of flight management
> tasks with functional categories and operational scenarios. IIFD3.1.2
> is for analytical human/machine tradeoff studies for functional
> categories to compare performance scores for both human and machine
> for each of the cells in the database produced by IIFD3.1.1. IIFD3.1.3
> is to define role-and-context-dependent function allocation strategies
> based on factor analyses of performance database and of pilot/operator
> pair-wise comparisons. IIFD3.1.4 is to develop and implement prototype
> function allocation schemes based on context detection ability
> (decision point for future research).*

### Risk analysis

Risk analyses should be performed to predict the probability of errors
in human-automation interaction. Fault-tree, event-tree and
cause-consequence diagrams can be applied to cognitive task analyses;
much as has been done in the past to systems such as nuclear power and
chemical process plants. A newer approach to human-machine reliability
analysis called "resilience engineering" assumes that there will be
unpredictable and therefore unavoidable errors, and puts the emphasis on
making the system resilient to failure and recovery easy and reliable.

> *IIFD2.2.03 seeks to define requirements for the application of
> predictive hazard models, simulation tools and analysis capabilities
> to mishap recreation. IIFD2.2.05 is for automatic identification of
> ten most frequently reported safety issues and vulnerabilities
> revealed through analysis of national of safety data archives*[^3]*;
> results are to be verified by domain experts from operational pilot
> community and air carrier safety groups. IIFD2.2.7 requires
> identification and prioritization of events and trends that could
> compromise system-wide safety due to new flight deck concepts.
> IIFD2.2.9 wants to deliver principles for the design of integrated
> computation, logic and simulation-based prediction tools for mishap
> re-creation.*

# PRIORITY RESEARCH NEEDS

Below are what the editors believe to be the highest priority research
needs:

## Performance of Humans Interacting with Automation and Optimization Processes

Since the viability of the NGATS system relies almost entirely on
surpassing unaided human performance limits and augmenting or changing
the bottlenecks presumed imposed by human performance, the primary set
of research issues concerns human-automation interaction. Can pilots
manage multiple alternatives in support of optimization? How many
options should be presented, and on what basis should the operators'
decisions be made? How are attention allocation strategies managed in
monitoring automation? How do the operators know when to attend to
optimization modification, and if they are alerted for input, what form
should that alert take? What are the default conditions in the
optimization process? How do the operators know what they are? How are
known biases toward over-reliance to be avoided? How are fail-soft modes
and resilience built into the system? How is reversion to operational
levels that can be managed by the human operators assured?

> *Milestone AS1.2.02 seeks to synthesize human factors and operational
> literature to identify limits of human performance in managing many
> aircraft. AS2.5.09 considers human performance broadly in the context
> of separation assurance. IIFD1.8.01 and IIFD1.8.04 are to model
> operator monitoring requirements and failure modes.*

## Single and Mixed-Mode Communication and Negotiation

The second major change in the NGATS paradigm relative to the current
NAS is the free flow of many types of information from many sources to
support the human-automation control. Therefore the issues of
human-human and human-automation communication are very critical. How
are human-to-human and human-to-decision-support-tool
communication/coordination conducted? If they are conducted in a mixed
voice and digital mode (through either textual or visual images), how
are the procedures and protocols of that interaction to be managed? What
special terms or procedures will assure consistent and accurate
coordination?

> *Milestone IIFD2.2.01 and IIFD3.2.02 seek to specify flight deck
> information requirements. Examples of milestones aimed at specifying
> information requirements are mentioned in many of the previous
> sections.*

## Individual and Team/Group Situation Awareness Issues

Given the dynamic, responsive, and information-intensive nature of the
operation of the NGATS system, there is a need to support and assure the
necessary levels of awareness of the operating state of the system,
awareness of individual areas of responsibility, and awareness of the
impact of individual actions on the system as a whole. This research
would seek to extend the situation awareness paradigms from individual
to collective awareness: it applies this paradigm to the issues raised
above in the net-centric discussions.

> *Milestone AS2.2.02 intends work with industry and the JPDO to model
> situation awareness in control, navigation and surveillance. AS3.5.06
> seeks to establish human-automation cooperation in separation
> assurance. Examples of milestones to assess situation awareness are
> mentioned in many of the previous sections.*

# APPENDIX A. QUESTIONS POSED TO PARTICIPANTS {#appendix-a.-questions-posed-to-participants .unnumbered}

The following questions were posed to the participants prior to the
workshop:

Air-ground integration and collaborative decision-making, including:

-   Evaluator/other automation proposes an action, route or clearance,
    pilots and controllers

> accept or negotiate that suggestion. Can that work?

-   How do pilot/controller actually do the evaluation of the proposed
    ("optimized") plan?

-   Problems with data-link as the principal communication medium, voice
    as exception?

-   How are negotiations ended?

-   Time constraints for decisions and handoffs? Who has final authority
    (between humans and automation and between humans), and under what
    circumstances?

Structuring of net-centric information, including:

-   What should be "pushed" (necessarily displayed to pilot or
    controller) under what contingencies?

-   What should have to be "pulled" (requested by pilot or controller),
    and how?

-   What should be limited (how much information is too much) and under
    what contingencies?

ATM workforce selection and training, including:

-   Requirements for different selection standards from present?

-   Workforce size requirements based on workload and scheduling in a
    restructured organization?

-   Changes in training/certification/requirements: how to ensure safety
    and acceptance?

Transition planning to 2025, including:

-   What human-automation issues will take how much time to research to
    provide satisfactory design/procedure development guidance?

-   Strategy for beginning paradigm shift to data-link, ADS-B,
    Evaluator, etc. Can it or should it evolve?

-   Problems when computer-based decision support tools result in being
    used other than as planned (example URET)?

-   How to improve the safety culture of NGATS as compared to present
    system. Can widely available performance data obviate the "blame
    game"?

Adequacy of human-automation research methodologies to support NGATS:

-   Where are the existing human-automation research methods solid and
    where weak, etc?

```{=html}
<!-- -->
```
-   Task analysis (both cognitive and functional)

-   Quantitative analysis of information and effects of time delay in
    outer control loops

-   Human-in-the-loop simulation: how much fidelity at what point in the
    development?

-   Fast-time simulation using human-system models

-   Prospective analysis of human errors, automation failures, and
    probabilistic risk analysis

-   Other approaches

# APPENDIX B. ATTENDEES {#appendix-b.-attendees .unnumbered}

Aguila, Frank (NASA) \<faguila@mail.arc.nasa.gov\>

Allard, Terry (NASA) \<terry.allard@nasa.gov\>

Andrews, John (MIT Lincoln Lab) \<jandrews@ll.mit.edu\>

Arbuckle, Doug (JPDO) \<<p.d.arbuckle@nasa.gov>\>

Ashford, Rose (NASA) \<rosemary.a.ashford@nasa.gov\>

Barhydt, Richard (NASA) \<<richard.barhydt@nasa.gov>\>

Campbell, Alan (ALPA) \<ga268@bellsouth.net\>

Cavolowsky, John (NASA) \<<jcavolowsky@mail.arc.nasa.gov>\>

Conway, Sheila (NASA) <sheila.r.conway@nasa.gov>

Consiglio, Maria

Corker, Kevin (San Jose State Univ.) \<kevin.corker@sjsu.edu\>

Davis, David (VGO Associates) \<<ddavis@vgoassociates.com>\>

Galushka, Joe (FAA) \<joe.galushka@faa.gov\>

Goodrich, Ken (NASA) \<kenneth.h.goodrich@nasa.gov\>

Hiremath, Urmila (MITRE) \<<urmila@mitre.org>\>

Jehlen, Richard  (FAA) \<richard.jehlen@faa.gov\>

John, Richard (Volpe) \<john@volpe.dot.gov\>

Jones, Patty (NASA) \<<pmjones@mail.arc.nasa.gov>\>

Kirkman, Debbie (MITRE) \<dkirkman@mitre.org\>

Latorella, Kara (NASA) \<k.a.latorella@nasa.gov\>

Lebecqz, Victor (U.C. Santa Cruz) \<<jvlebacq@ucsc.edu>\>

Manning, Carol (FAA) \<carol.manning@faa.gov\>

Mogford, Richard (NASA) \<<richard.h.mogford@nasa.gov>\>

Murdoch, Jennifer [(NASA)
\<jennifer.l.murdoch@nasa.gov](mailto:(NASA) <jennifer.l.murdoch@nasa.gov)\>

Nadler, Eric (VOLPE) \<<nadler@volpe.dot.gov>\>

Overbey, Chuck (FAA) \<charles.overbey@faa.gov\>

Parasuraman, Raja (George Mason Univ.) \<rparasur@gmu.edu\>

Poage, Jim (JLP Performance Consulting) \<poage@post.harvard.edu\>

Pritchett, Amy (Georgia Tech) \<amy.pritchett@ae.gatech.edu\>

Quon, Leighton (NASA) \< leighton.quon@nasa.gov\>

Sarter, Nadine (Univ. of Michigan) \<sarter@umich.edu\>

Sheridan, Tom (Volpe) <sheridan@volpe.dot.gov>

Townsend, Brian (ALPA) \<brian.townsend@alpa.org\>

White, John (NASA) \<john.j.white@nasa.gov\>

Wing, David (NASA) \<<david.j.wing@nasa.gov>\>

Woldring, Michael (Eurocontrol) \<michael.woldring@eurocontrol.int\>

Zellweger, Andres (JPDO) \<azellweg@hq.nasa.gov\>

# ACKNOWLEDGMENTS

Dr. Richard John of the Volpe Center was instrumental in initiating this
project and contributing ideas throughout. Dr. James L. Poage of JLP
Performance Consulting provided valuable assistance in reviewing
participants' commentary and preparing this report. Dr. Andres Zellweger
of JPDO helped with arrangements in the Washington DC area. Sunnie Neal
and William Eagar of the Georgia Institute of Technology Research
Institute kindly offered their excellent meeting facilities and
hospitality.

[^1]: Next Generation Air Transportation System (NGATS) Air Traffic
    Management (ATM)-Airspace Project Reference Material. Submitted June
    1, 2006. External Release Version. Delivered by Harry Swenson,
    Richard Barhydt, and Michael Landis.

[^2]: Aviation Safety Program. Integrated Intelligent Flight Deck.
    Technical Plan Summary. Dr. Steven D. Young, and Leighton Quon (ND).

[^3]: Archives include [Flight Safety Foundation Flight Operational
    Quality Assurance,](http://www.flightsafety.org/foqa.html) NASA/FAA
    [Aviation Safety Reporting System](http://asrs.arc.nasa.gov/), and
    NASA Aerospace Safety Advisory Panel /Aviation Safety.