NASA/TM—2011–215997

Metrics for Operator Situation Awareness, Workload, and 

Performance in Automated Separation Assurance Systems

Thomas Z. Strybel and Kim-Phuong L. Vu

California State University, Long Beach

Vernol Battiste and Arik-Quang Dao

San Jose State University

John P. Dwyer

The Boeing Company

Steven Landry

Purdue University

Walter Johnson

NASA Ames Research Center

Nhut Ho

California State University, Northridge

December 2011
ii

NASA STI Program...in Profile

Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space 

science. The NASA scientific and technical 

information (STI) program plays a key part in 

helping NASA maintain this important role.

The NASA STI program operates under the 

auspices of the Agency Chief Information 

Officer. It collects, organizes, provides for 

archiving, and disseminates NASA’s STI. The 

NASA STI program provides access to the 

NASA Aeronautics and Space Database and its 

public interface, the NASA Technical Report 

Server, thus providing one of the largest 

collections of aeronautical and space science 

STI in the world. Results are published in both 

non-NASA channels and by NASA in the 

NASA STI Report Series, which includes the 

following report types:

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant 

phase of research that present the results of 

NASA Programs and include extensive data 

or theoretical analysis. Includes 

compilations of significant scientific and 

technical data and information deemed to be 

of continuing reference value. NASA 

counterpart of peer-reviewed formal 

professional papers but has less stringent 

limitations on manuscript length and extent 

of graphic presentations.

• TECHNICAL MEMORANDUM. 

Scientific and technical findings that are 

preliminary or of specialized interest, e.g., 

quick release reports, working papers, and 

bibliographies that contain minimal 

annotation. Does not contain extensive 

analysis.

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored 

by NASA.

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest.

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to 

NASA’s mission.

Specialized services also include creating custom 

thesauri, building customized databases, and 

organizing and publishing research results.

For more information about the NASA STI 

program, see the following:

• Access the NASA STI program home page at 

http://www.sti.nasa.gov

• E-mail your question via the Internet to 

help@sti.nasa.gov

• Fax your question to the NASA STI Help 

Desk at (301) 621-0134

• Phone the NASA STI Help Desk at 

(301) 621-0390

• Write to:

NASA STI Help Desk

NASA Center for AeroSpace Information

7121 Standard Drive

Hanover, MD 21076-1320
iii

NASA/TM—2011–215997

Metrics for Operator Situation Awareness, Workload, and 

Performance in Automated Separation Assurance Systems

Thomas Z. Strybel and Kim-Phuong L. Vu

California State University, Long Beach

Vernol Battiste and Arik-Quang Dao

San Jose State University

John P. Dwyer

The Boeing Company

Steven Landry

Purdue University

Walter Johnson

NASA Ames Research Center

Nhut Ho

California State University, Northridge

National Aeronautics and Space Administration

Ames Research Center

Moffett Field, California

December 2011
iv

The use of trademarks or names of manufacturers in this report is for accurate 

reporting and does not constitute an official endorsement, either expressed or 

implied, of such products or manufacturers by the National Aeronautics and 

Space Administration

Available from:

NASA Center for Aerospace Information

7121 Standard Drive

Hanover, MD 21076-1320

301) 621-0390

This report is also available in electronic form at http://www.sti.nasa.gov 

or http://ntrs.nasa.gov/
v

Table of Contents

List of Acronyms and Definitions...................................................................................... vi

Executive Summary............................................................................................................ 1

1. Introduction .................................................................................................................... 3

1.1 Objectives............................................................................................................... 3

1.2 Approach ................................................................................................................ 3

2. Method Development ..................................................................................................... 5

2.1 Distributed Simulation Capability.......................................................................... 5

2.2 Metric Development............................................................................................... 6

3. Systems Analysis of NextGen Concepts of Operation................................................... 7

3.1 Evaluation of NextGen Concepts of Operation...................................................... 7

3.2 Analysis of Information Requirements .................................................................. 8

3.3 Implications of NextGen Roles and Responsibilities for Operator Training ......... 9

4. Results ............................................................................................................................ 10

4.1 Part-Task Simulations and Experiments ................................................................ 10

4.1.1 Conflict Detection with NextGen Flight-Deck Tools ................................... 10

4.1.2 Metrics for Situation Awareness and Workload ........................................... 11

4.1.3 Alternative Approach to Situation Awareness Measurement and

Validation: A Set Theoretic Framework ....................................................... 12

4.2 Distributed Air-Ground Simulations...................................................................... 14

4.2.1 Trajectory Oriented Operations with Limited Delegation during

Convective Weather (TOOWiLDx) Simulation Demonstrations ................. 14

4.2.2 Evaluation of Automated Spacing Support Tools for Interval

Management Operations under Hazardous Weather Conditions .................. 14

4.2.3 Situation Awareness with Trajectory Oriented Operations in

Weather (SAWTOOth).................................................................................. 15

5. Conclusions .................................................................................................................... 16

6. Publications and Presentations Citing the NRA............................................................. 17

7. References ...................................................................................................................... 22
vi

Acronyms and Definitions

4D-CSD...............Four-Dimensional Cockpit Situation Display

ADRS ..................Aeronautical Data Link and Radar Simulator (NASA Ames Research Center)

ADS-B.................Automatic Dependent Surveillance – Broadcast

ASAS...................Automated Separation Assurance Systems

ATC.....................air traffic control

ATM ....................air traffic management

AT-SAT...............Air Traffic Selection and Training

CDA.....................continuous descent approach

CHAAT ...............Center for Human Factors in Advanced Aeronautic Technologies (California 

State University, Long Beach)

CSULB ................California State University, Long Beach

CSUN ..................California State University, Northridge

FDDRL................Flight Deck Displays Research Laboratory (NASA Ames Research Center)

HSEL...................Human Integrated Systems Engineering Laboratory (Purdue University)

IAS.......................indicated air speed

JPDO....................Joint Planning and Development Office

LOS......................loss of separation

MACS..................Multi Aircraft Control System

NASA ..................National Aeronautics and Space Administration

NextGen...............Next Generation Air Transportation System

NRA.....................NASA Research Annouoncement

RAT.....................route assessment tool

SA .......................situation awareness

SAGAT................Situation Awareness Global Assessment Technique

SAWTOOth II .....Situation Awareness in Trajectory Oriented Operations with Weather II

SDF......................Louisville International Airport

SERL ...................Systems Engineering Research Laboratory (California State University, 

Northridge)

SJSUF..................San Jose State University Foundation

SPAM ..................Situation Present Assessment Method

ZID ......................Indianapolis Center

ZKC.....................Kansas City Center
1

Executive Summary

A research consortium of scientists and engineers from California State University Long Beach 

(CSULB), San Jose State University Foundation (SJSUF), California State University Northridge 

(CSUN), Purdue University, and The Boeing Company was assembled to evaluate the impact of 

changes in roles and responsibilities and new automated technologies, being introduced in the Next 

Generation Air Transportation System (NextGen), on operator situation awareness (SA) and 

workload. To meet these goals, consortium members performed systems analyses of NextGen 

concepts and airspace scenarios, and concurrently evaluated SA, workload, and performance 

measures to assess their appropriateness for evaluations of NextGen concepts and tools. The 

following is a summary of our activities and accomplishments that were supported by the NASA 

Research Announcement (NRA).

• Distributed Simulation. Concept and metric evaluation required that we develop a 

distributed simulation network in which each laboratory participated in large-scale 

simulations of NextGen concepts. With this distributed-simulation foundation established, 

two distributed-simulation demonstrations and two distributed-simulation experiments were 

completed in year 3 of the NRA. 

• Metric Development. Based on our review of SA and SA measurement, we determined that 

probe techniques are the most promising measures for evaluating the impact of NextGen 

changes in roles and responsibilities on operator SA and workload. The probe techniques 

have been shown to have relatively good psychometric properties, especially predictive 

validity and diagnosticity. We modified the method developed by Durso (e.g., Durso and 

Dattel, 2004) by automating SA probing, improving on the method of presenting probe 

questions and collecting operator responses, working toward a standard set of probe 

questions (or probe question categories) and adding workload queries. Several simulations 

were run on both pilots and air traffic controllers (ATCs). These provided evidence for the 

validity, sensitivity and diagnosticity of the method. 

• Systems Analysis. System analyses of potential NextGen concepts of operation were also 

reviewed. We determined that these concepts could be distributed along two axes: the 

degree to which responsibility for separation assurance and collision avoidance is assigned 

to the controllers versus the pilots, and the degree to which automation augments or replaces 

controller and pilot functions. Interviews with air traffic controllers and commercial pilots 

for task goals information requirements were also done. ATCs were also interviewed about 

training requirements for NextGen operators. ATCs responded that the next generation of 

air traffic controllers would not benefit much from previous experience with computers and 

games because the perceptual motor skills are not the most important skill involved in air 

traffic management.

• Part-Task Simulations. A series of part-task simulations and experiments were conducted 

within each consortium organization. These were focused on conflict detection and 

resolution in NextGen, development of metrics for SA and workload in NextGen, 

application of the metrics to NextGen airspace environments, and the evaluation of 

NextGen concepts of operation for separation assurance. These were followed by a series of 

distributed air-ground simulations that involved simulation roles (flight decks, controllers, 
2

pseudo-aircraft, etc.) being played by two or more member organizations. Part-tasksimulation experiments directed at metric development established the validity, sensitivity 

and diagnosticity of the modified online probe technique. Experiments on pilot conflict 

resolution showed that pilots were not comfortable accepting automated conflict resolutions 

and desired to communicate with ATCs more than is anticipated in NextGen.

• Large-Scale Simulations. Our distributed simulations evaluated plausible NextGen concepts 

of operations, and cockpit automation tools for traffic separation, weather avoidance, 

merging and spacing, and communications. These projects simulated sectors in Kansas City 

Center (ZKC) and Indianapolis Center (ZID), with participant pilots flying en route, arrival 

and approaches to Louisville International Airport (SDF), while avoiding weather and 

managing spacing. Three concepts of operation were evaluated that allocated primary 

responsibility for traffic separation between pilots, ATCs and automation. We determined 

that SA was highest when pilots were responsible for traffic separation. Workload was 

unaffected by adding responsibility for traffic separation. Workload was only affected by 

task demands related to flight phase. Weather perturbations did create situations in which 

aircraft spacing was negatively affected regardless of whether the pilot faithfully followed 

the leading aircraft around weather.

As a result of this NRA, we have gained a greater understanding of SA and its measurement, and 

have shared our knowledge with the scientific community: During the NRA, we published 20 

technical papers, made 21 presentations at conventions, presented 6 briefings to the National 

Aeronautics and Space Administration (NASA), and prepared 5 unpublished reports. Additional 

technical papers based on the NRA work were also published after the funding period. Through the 

success of our distributed-simulation network, we have shown how to create a flexible and costeffective test bed for simulations and research involving current and future airspaces. This network 

provides a mechanism for consortium members, colleagues, and students to pursue research on 

other topics in air traffic management and aviation, thus enabling them to make greater 

contributions to the field. 
3

1. Introduction

1.1 Objectives

Introducing new air traffic management (ATM) concepts, Automated Separation Assurance Systems 

(ASAS), and other automation technologies proposed by Next Generation Air Transportation System

(NextGen) can significantly impact workload and situation awareness (SA) for operators in future 

airspace systems. Successful implementation of these solutions, for achieving Joint Planning and 

Development Office (JPDO) goals, demands that we identify the information required of each 

operator in performing his/her new role, and the impact of new technologies on operator SA and 

workload. As a result, current metrics need to be revised or new measurement techniques need to be 

developed to ensure that they are reliable, valid, and sensitive to the changes in operator SA and 

workload brought about by NextGen innovations. The objectives established for our NASA Research 

Announcement (NRA) are as follows: 

• Develop methods for quantifying required and actual operator SA, workload, and 

performance in relation to:

- Operator (flight crew and controller) role in Automated Separation Assurance 

Systems

- The impact of these factors on system safety and performance under NextGen

• Characterize individual and shared SA in NextGen environments.

• Support program capability for human-in-the-loop simulations of NextGen concepts and 

evaluations of SA, workload, and performance.

1.2 Approach

A research consortium of scientists and engineers from California State University Long Beach

(CSULB), San Jose State University Foundation (SJSUF), California State University Northridge

(CSUN), Purdue University, and The Boeing Company was assembled. Our approach to the 

development and evaluation of SA, workload, and performance was characterized by systems 

analysis (conducted by systems engineers and operations experts, with inputs from NASA) of 

NextGen concepts and airspace scenarios. Concurrently, metric evaluations of existing measures of 

SA, workload, and performance were conducted to assess their ability to predict SA in NextGen 

environments. These parallel approaches drove the design of simulations to evaluate candidate 

measures for their ability to reflect changes in operator SA, workload, and performance that might be 

induced by NextGen concepts.

Our multidisciplinary team of universities and private industry established the technological 

capacity to conduct networked, human-in-the-loop simulations for assessing candidate SA metrics, 

workload, and performance for individual pilots and air traffic controllers (ATC), pilot-controller 

teams, and pilot-controller interactions with ASAS. All members of the consortium have extensive 

knowledge and experience in the areas of human performance and ATM operations. Table 1 

describes the member organizations in terms of the institution, key personnel, and the role of each 

team member in the NRA.
4

Table 1. Consortium Organizations and Descriptions of

Principal Investigators (PI) and Co-Investigators (CI)

California State University, Long Beach; Center for Human Factors in 

Advanced Aeronautic Technologies (CHAAT)

• Thomas Strybel, PI; Professor of Psychology and Director of 

CHAAT

• Kim-Phuong L. Vu, Co-I; Associate Professor of Psychology and CoDirector of CHAAT

San Jose State University Foundation

• Vernol Battiste, Co-I; Senior Research Associate for the FDDRL at 

NASA Ames Research Center

• Arik-Quang V. Dao, Co-I; Research Associate for the FDDRL

Purdue University, Human Integrated Systems Engineering Laboratory 

(HSEL)

• Steven Landry, Co-I; Assistant Professor of Industrial Engineering 

and Director of HISEL

California State University, Northridge; Systems Engineering Research 

Laboratory (SERL)

• Nhut Ho, Co-I; Associate Professor of Mechanical Engineering and 

Director of SERL

The Boeing Company

• John P. Dwyer, Co-I; Technical Fellow, Human Systems Integration

By taking advantage of the unique strengths and capabilities of each member the consortium, we made 

significant progress in achieving the goals of the NRA. Our objectives were accomplished through: 

• Part task, in-house simulations conducted individually at consortium laboratories

• Distributed simulations involving two or more consortium organizations

• Laboratory experiments conducted individually within the consortium organizations

• Systems analyses and concept papers developed by one or more Co-Investigator.

Our accomplishments with respect to the objectives of this NRA are detailed in the subsequent pages.

They have been organized into three major sections. Section 2 summarizes the methods we developed 

for distributed simulation capability, and SA and workload measurement. Section 3 summarizes 

systems analyses we conducted of operator roles and responsibilities in NextGen, information 

requirements and implications of NextGen concepts on operator training. Section 4 summarizes the 

results obtained from application of the methods, the knowledge we have obtained regarding how to 

measure SA, workload, and performance in NextGen air traffic environments, and evaluations of 

NextGen concepts in terms of changes in operator roles and their effect on operator SA, workload, 

and performance.
5

2. Method Development

2.1 Distributed Simulation Capability 

Our consortium members have established a simulation network infrastructure that permits 

interaction between controllers, pilots, and experimenters over the Internet. This capability is made 

possible through the implementation of software donated to the consortium members by NASA Ames 

Research Center: Aeronautical Data Link and Radar Simulator (ADRS), Multi Aircraft Control 

System (MACS), Four Dimensional Cockpit Situation Display (4D-CSD), and Distributed 

Air/Ground Voice communication. Taking advantage of both the existing infrastructure at each 

consortium site and the adaptability of NASA Ames’ simulation software, we created an extremely 

flexible simulation network. Building on NASA’s own plan for a versatile simulation environment, 

our distributed simulation capability enabled us to readily host large simulations that can vary the 

number of self-separation–capable aircraft, simulate current day, NextGen, or mixed-equipage 

airspace environments, and assess how different levels of automation associated with different 

NextGen concepts of operations affect operator and system performance.

The distributed simulation capability was successfully demonstrated in 2008. First, we established 

connectivity and ensured adequate bandwidth at each facility. Then we installed simulation software 

and trained personnel at each facility on simulation software and procedures, thus expanding the 

capacity for airspace simulations in one facility (CHAAT at CSULB) and creating two additional 

simulation facilities in consortium laboratories (SERL at CSUN and HISEL at Purdue). Once 

operational, FDDRL engineers improved the distributed simulation network by creating a capability 

for remote startup of all simulation stations to ensure that all stations at each facility were 

appropriately connected and configured before the start of a simulation run. With this distributed 

simulation foundation established, two distributed-simulation demonstrations and two distributedsimulation experiments were completed in year 3 of the NRA. Table 2 shows the distributed 

simulation configuration for one of the simulation experiments, the Situation Awareness in Trajectory 

Oriented Operations with Weather II (SAWTOOth II) simulation that was conducted between August 

and September 2009, described in Section 4.2. This simulation involved over 30 workstations at four 

consortium laboratories.

Table 2. Distributed Simulation Components for the Situation Awareness

and Workload in Trajectory Oriented Operations with Weather Simulation*

FDDRL

(NASA Ames 

Research Center)

CSAAT 

(Calif. State Univ., Long 

Beach)

HISEL

(Purdue Univ.)

SERL 

(Calif. State Univ., 

Northridge)

• 6 participant pilot

stations

• 5 single pseudopilot

stations

• 1 777 simulator

• 1 eye-tracking 

station

• 1 Simulation 

Manager station

• 1 Voice Server

• 1 ADRS station

• 2 participant ATC 

stations

• 2 ghost/confederate 

ATC stations

• 8 multi-pilot stations

• 2–4 probe stations

• 1–2 ADRS stations

• 1 ADRS station

• 2 single pseudopilot 

stations

• 2 single 

pseudopilot

stations

• 1 ADRS station

* See Section 4.2.
6

In summary, our development of a distributed simulation capability not only created a mechanism for 

running large-scale simulations of most NextGen operator roles, but also expanded the number of 

universities capable of performing NextGen airspace simulations.

2.2 Metric Development 

NextGen innovations must be developed using a system-comprehensive, user-centered strategy that 

promises continuing improvement in efficiency and maintenance—and even enhancement—of safety.

However, NextGen automation solutions may alter operator SA and workload in negative ways, and 

it is important that valid and reliable measures be available for evaluating SA, workload, and 

performance in NextGen. As part of the NRA, we evaluated existing metrics, and developed new 

techniques, for quantifying operator SA in current day and NextGen aviation environments.

We first reviewed existing metrics of SA, workload, and performance, and evaluated each metric for 

metric properties of reliability, validity, sensitivity, diagnosticity, usability, and operator acceptance.

Based on our review of SA and SA measures, we determined that probe techniques are the most 

promising measures for evaluating the impact of NextGen changes in roles and responsibilities on 

operator SA and workload. The probe techniques have been shown to have relatively good 

psychometric properties, especially predictive validity and diagnosticity (e.g., Endsley, 1990; Endsley 

& Smolensky, 1998). With regard to existing probe techniques, we determined that real-time probe 

methods were more advantageous than freeze-probe methods because real-time probes are not 

heavily dependent on working memory and can assess SA when operators off-load information to the 

environment (Chiappe, Strybel, & Vu, in press). We also determined that some issues related to the 

existing real-time probe technique, the Situation Present Assessment Method (SPAM), needed to be 

addressed to make the method applicable for NextGen investigations of SA and workload. In 

particular, we determined that the impact of workload must be clearly identified in the use of online 

probes because workload can impact operator performance independent of SA. Relating to this point, 

we ran several studies that examined how workload and other factors affected SA measures using the 

on-line probe technique. We found:

• Latencies to SA probes can be influenced by the response input method. Therefore, we 

streamlined the administration of probes from free-responses to multiple-choice and 

true/false or yes/no format, to reduce the added workload produced by the query task. We 

also improved the apparatus for presenting queries and collecting operator responses.

• We determined that a standard method for developing probe queries needed to allow 

comparison of SA metrics across simulation studies. In our approach, queries were created 

beforehand in conjunction with scenario development. Subject matter experts were 

consulted, but we also created queries based on categories relevant to SA: processing level, 

time frame, and information content. After several part task simulations, we determined that 

questions should be based on information requirements. Moreover, once categories are 

established, we found that it was important to design probes so that each combination of 

categories is probed an equal number of times and the order in which information queries 

are presented must be counterbalanced across scenarios and participants.

We ran several simulations for refining and validating our online probe technique. The technique is 

based on Durso’s SPAM technique (Durso & Dattel, 2004; see Figure 1). To ensure that response 

latency was related to SA and not workload, each query begins with a “Ready” prompt and audio 

alert. Operators were instructed to respond “yes” to the ready prompt only when they had sufficient 

time to take a SA question. If the operator, for example, a pilot, responded affirmatively to the ready 

response, the probe question was immediately presented and the pilot responded by selecting the 
7

answer (typically yes/no or multiple choice). The results of our metric-development work, including a 

description of the administration and question development techniques were documented in a Manual 

for Online Probing that was submitted to our NASA Technical Monitor in January 2011 (Strybel et 

al., 2011). Results of simulations using the real-time probe technique are summarized in Section 4.

Figure 1. Assumptions and example (pilot query) of our real-time SA and workload probe 

technique.

3. Systems Analysis of NextGen Concepts of Operation

Analyses of the information requirements for pilot and ATC tasks under different NextGen function 

allocation and automation concepts was performed in years 1 and 2 of the NRA. Because a clear 

picture of the most likely NextGen solutions was not yet available, we surveyed the views of NASA 

scientists on the proposed NextGen solutions and determined the impact that each solution would 

have on operator roles and responsibilities, workload, and SA. Interviews with pilots and ATCs were 

also conducted to determine the impact of specific changes in roles and responsibilities on the 

information and training requirements of future operators. 

3.1 Evaluation of NextGen Concepts of Operation

In parallel with the information requirements analyses, we also conducted a review of separation 

assurance and collision avoidance operational concepts for NextGen. The review showed that the 

concepts can be distributed along two axes: the degree to which responsibility for separation 

assurance and collision avoidance is assigned to the controllers versus the pilots, and the degree to 

which automation augments or replaces controller and pilot functions. Based on an analysis of the 

implications of these concepts to NextGen, from a human factors standpoint as well as the 

technological readiness of the concepts, it was concluded that some form of supervisory control of 

separation assurance by controllers is the most viable concept. For additional details see Table 3 and 

Dwyer and Landry (2009).
8

Table 3. Taxonomy of Separation Assurance Concepts and 

Projected Impact on Situation*

Expected Impact Comparing NextGen Environment

to Current Environment

Concept

Expected 

technological 

readiness

ATC 

situation

awareness

ATC 

workload

Pilot 

situation 

awareness

Pilot 

workload

Concept 

appears 

viable?

Shared 

responsibility 

no automation

High Small 

reduction Excessive Moderate increase Excessive No

ATC 

automation

Conflict ID 

only Medium-high Neutral Excessive Neutral Neutral No

Conflict ID 

with resolution

tools

Medium Neutral

Very 

large 

increase

Neutral Neutral Yes

Conflict ID 

with resolution

options

Medium Small 

reduction

Moderate 

increase Neutral Neutral Yes

Conflict Id 

with autoresolver

Medium Large 

reduction

Moderate 

reduction Neutral Neutral Yes

Distributed 

control Low Very large reduction

Large 

reduction

Large 

increase

Large 

increase No

Mixed 

concepts

Medium-low Moderate 

reduction

Moderate 

increase

Moderate 

increase

Moderate 

increase

Yes

* Dwyer and Landry, 2009.

3.2 Analysis of Information Requirements

To support the evaluation of feasible function allocation concepts for separation assurance systems, 

and to develop a better understanding of the specific information requirements for key tasks 

(resolving conflicts, avoiding weather, and merging and spacing), air traffic controllers and 

commercial pilots were interviewed for their goals, sub-goals, and the individual and shared 

information needed to perform the tasks. For high-level goals, the two pilots interviewed were in 

general agreement in terms of their responses, even though they flew for different airlines. The two 

controllers interviewed were also in consensus. However, for the same high-level goals, the sub-goals 

of the pilots and the controllers showed more differences than similarities. For example, in the 

conflict resolution task, the effect of allocating separation responsibility to pilots (Concept 1), ATCs 

(Concept 2) and Automation (Concept 3) is shown in Table 4. Pilots based their decision for 

trajectory changes on factors such as fuel consumption, time to destination, secondary conflicts, and 

passenger comfort. The controllers, on the other hand, based their decisions on factors such as the 
9

effect that the trajectory change has on the entire traffic flow, additional conflicts induced, and 

workload. Thus, while the controllers and the pilots shared common goals at the highest level, the 

controllers’ motivations are system-centric, while the pilots are aircraft-centric. This point was further 

reinforced in the group interview session in which the controllers confirmed that they rarely take into 

account the sub-goals that are important to the pilots, while the pilots indicated that they prefer the 

controllers to take into account as much as possible the sub-goals that are important to them. The key 

information requirements obtained can be used as input to ascertain which information is most 

needed when developing probe questions for measuring individual and shared SA. The knowledge 

elucidated also provided insights into the interaction among the controllers, pilots, and automation, 

and their perception of the feasibility of different concepts for separation assurance.

Table 4. Pilot and ATC Goals and Sub-goals Across

Function Allocation Concepts for Resolve Conflict Task*

Concept 1 Concept 2 Concept 3

PILOT

Goals • Aware of conflicts near 

and far

• Avoid secondary 

conflict

• Avoid conflict • Avoid conflict

Sub goals • Fuel management

• Time management

• Manage alt/head

• Communicate with 

ATC

• Fuel management

• Passenger comfort

• Communicate with 

ATC

• Fuel management

• Passenger comfort

• Communicate with ATC

ATC

Goals • Separation • Separation • Separation

Sub goals • Reduce workload

• Communicate less

• Sector traffic

• Fuel levels for 

sequence priority

• Reduce workload

• Communicate less

• Sector traffic

• Fuel levels for 

sequence priority

• Reduce sector 

workload

* Ho, Martin, Bellissimo, and Berson, 2009.

3.3 Implications of NextGen Roles and Responsibilities for Operator Training

Dwyer, Gershzohn and Thorpe (2011) conducted interviews of ATCs who participated in a 

distributed simulation of potential NextGen Concepts of Operation (see Section 4.2) to assess their 

opinions on current and future training needs. Although ATC participants were initially skeptical of 

some concepts (i.e., allocated separation responsibility to the flight deck), once additional 

modifications were made to the procedures being simulated, and the ATCs were able to practice the 

concepts, they were more accepting of these concepts of operation. Dwyer et al. also asked ATCs 

about how NextGen tools will affect the training of future controllers, specifically with respect to the 
10

high computing and gaming backgrounds of this generation of controllers. The majority of ATCs 

indicated that a computer gaming background might improve perceptual-motor skills, but these are 

not the critical factors determining success. In NextGen, it is likely that ATCs will be working with 

automation in the course of conducting sector-level traffic flow management tasks. Moreover, one 

ATC noted that attrition rates for trainees (which were estimated at 50%) are typically the result of 

operational errors and conducting unsafe separation operations. Perceptual-motor skills most likely 

would not contribute to these errors.

4. Results

4.1 Part-Task Simulations and Experiments

A series of part-task simulations and experiments were conducted within each consortium 

organization to meet specific NRA objectives. These studies were focused on the conflict detection 

and resolution in NextGen, development of metrics for SA and workload in NextGen, application of 

the metrics to NextGen airspace environments, and the evaluation of NextGen concepts of operation 

for separation assurance.

4.1.1 Conflict Detection with NextGen Flight-Deck Tools

An airside part-task study was designed and run individually at NASA Ames’ FDDRL and three 

consortium laboratories: CSULB, CSUN, and Purdue. Flight crew responses to ground-side 

automation-derived conflict resolutions were evaluated under three flight-deck decision-aiding 

modes: automated aiding, interactive aiding, and manual resolution. In the automated aiding 

condition, pilots executed all resolutions generated by the automation. In the interactive condition,

automation suggested a maneuver, but pilots had the choice of accepting or modifying the provided 

resolution. In the manual condition pilots generated resolutions independently. Pilots’ acceptability of 

different types of conflict resolutions provided by the automation (vertical and horizontal) at different 

ranges (near and far) from ownship was measured. Pilot SA was also determined for each of the 

flight-deck decision-aiding modes. Findings from this study were summarized in Dao et al. (2009) 

and Battiste et al. (2008).

Conflict resolutions created either by the automation or by flight crews were safe; all resolutions 

maintained a separation distance greater than 5 nm. Crews rated approximately 30% of automated 

resolution as problematic and reported that they would seek ATC input. However, when allowed to 

modify automated resolutions with flight-deck route-planning tools, the crews wanted to consult with 

ATC on approximately 8% of the resolutions. Finally, crews reported that the decision to accept, 

reject, or modify an automated resolution is a complex and situation-dependent decision. When close 

to TOD they generally preferred to descend, but when 500 nm or more from TOD they generally 

preferred to climb. Situation awareness was measured with probe questions using a technique that 

was a combination of online- and freeze-probe methods. SA was higher in the manual- and 

interactive-aiding conditions compared with the automated condition, suggesting that pilots had 

higher SA when they were actively engaged in the conflict resolution task. It was also shown that 

pilots were more comfortable reviewing automated conflict resolutions, as well as modifying those 

resolutions before execution, compared to simply executing the resolution sent to them by the 

automated system.

4.1.2 Metrics for Situation Awareness and Workload

We performed several simulations to develop tools and methods for evaluating NextGen ATM 

concepts in terms of operator SA, workload, and performance, and the impact of these factors on 
11

system safety and performance. Although each simulation involved both ATCs and pilots, only one 

operator role was evaluated in each study. In each study we investigated the validity, reliability, 

sensitivity and diagnosticity of workload and SA probe latencies and accuracy.

Strybel, Vu, Kraft, and Minakata (2008) compared two methods of SA assessment with easy and hard 

self-spacing scenarios: a freeze-probe technique in which the scenario is frozen and pilots are 

administered a battery of SA questions, and an online-probe technique in which SA questions are 

administered while pilot is engaged in the primary task (flying). Performance was assessed with the 

mean and standard deviation of the indicated air speed (IAS), and number of missed ATC 

instructions. Freeze-Probe accuracy was marginally correlated with IAS variability, and online probe 

latency was significantly correlated with both IAS variability and missed ATC instructions.

Moreover, a significant correlation between post-scenario NASA TLX and SPAM ready Latency was 

obtained, which confirmed the assumption regarding the relationship between ready latency and 

workload. A second simulation provided additional support for the diagnostic capability of online 

probes (see Section 4.2).

The validity and diagnosticity of the online probe technique for ATCs was demonstrated in three 

part-task simulations. Strybel, Minakata, Nguyen, Pierce, and Vu (2009) examined the relative 

effectiveness of online questions designed to measure the types of processing required (recall, 

comprehension), time frame (present, future), and information/task content (sector status, conflicts, 

and command/communication). Both experienced (retired) ATCs and student ATCs were tested. The 

best types of probe questions queried operators about future conflicts and were in multiple-choice 

format. Latencies for probe questions that addressed conflicts significantly accounted for the variance 

in task performance measures (loss of separation [LOS], average vertical distance, and number of 

traffic advisories) related to safety.

Bacon et al. (2011) examined SA, workload, and performance of retired ATCs and student ATCs in a 

simulation of mid-term NextGen airspaces. Scenarios differed in the percentage of equipped 

(DataComm, ADS-B) aircraft. ATCs could mange equipped aircraft with NextGen tools (DataComm, 

conflict probe, and a trial-planner). Conflict alerts were provided only for conflicts between two 

equipped aircraft. To obtain an overall measure of ATC performance, each scenario run was 

recorded, and ATC performance was rated by three expert controllers using a modified FAA Air 

Traffic Selection and Training (AT-SAT) Over-the-Shoulder Rating form. The modifications allowed 

for ratings regarding the use of DataComm. Experienced ATCs resolved more non-alerted conflicts 

(fewer losses of separation) than student ATCs. Experts showed lower workload (shorter ready 

latencies) and higher SA (lower SA probe latencies) than students, especially for probes relating to 

conflict queries. SA probe latency and workload (ready) latency significantly accounted for variance 

in three AT-SAT Rating Dimensions: “Maintaining Attention and Situation Awareness,” 

“Maintaining Separation” and “Overall performance.” These findings provided additional evidence 

for the validity of an online SA and workload probe method.

Strybel et al. (2011) examined SA, workload, and performance of ATCs when the mixture of 

equipped and unequipped aircraft changed within a scenario and the change was accompanied by an 

increase in traffic, thus replicating a situation in which the percentage of equipped aircraft changes 

during an ATC’s work shift. ATC workload increased in the second half of the scenario presumably 

because of the increase in traffic. The number of LOSs between non-alerted aircraft was affected only 

by the number of conflicts. Other measures of ATC performance suggested that the strategies used 

for separating aircraft changed as a function of equipage change: When the percentage of equipped 
12

aircraft increased in the second half of the scenario, the mean vertical separation between aircraft 

increased and the mean lateral separation decreased, compared with scenarios in which the 

percentage of equipped AC either decreased in the second half or remained constant. Moreover, SA

of sector status information decreased (probe latencies increased) in the second half of scenarios in 

which the percentage of equipped aircraft decreased suggesting that SA was lower when an increase 

of unequipped aircraft was introduced into the scenario.

In summary, our work on metric development and evaluation, specifically with the online probe 

technique, can be summarized as follows:

• Situation awareness probe latencies are related to performance outcomes (LOS, conflict 

resolution time) that affect separation assurance for pilots and ATCs.

• Situation awareness probe latencies for pilots are sensitive to changes in responsibility for 

separation assurance and changes in scenario difficulty brought about by weather. Workload 

probe latencies were unaffected by changes in responsibility for traffic separation. However, 

pilot workload increased throughout the descent phase, and reached a peak when queried 

before the CHRCL (outer marker) waypoint (directly before landing). 

• Situation awareness probe latencies for ATCs are sensitive to scenario manipulations such as 

equipage, and number of conflicts. Probe latencies were shown to discriminate between 

student and expert performance in mixed equipage environments, consistent with expert 

AT-SAT ratings of overall performance, maintaining separation and maintaining attention 

and SA. Probe latencies for sector status information increased when the percentage of 

equipped aircraft increased, but latencies for conflict information was unchanged, 

suggesting that only awareness of status information was reduced when the percentage of 

equipped aircraft was reduced.

4.1.3 Alternative Approach to Situation Awareness Measurement and Validation: 

A Set Theoretic Framework

Landry and Surakitbanharn (2011) proposed a framework for assessing the relationship between SA

and performance based on identifying information requirements for good performance, and 

measuring operator recall of these using a freeze probe technique such as the Situation Awareness

Global Assessment Technique (SAGAT). Moreover, once information requirements are known, it 

should be possible to identify a “mapping function” that mathematically describes the relationship 

between performance and recall of required information. If mapping functions can be found that 

relate performance to recall using SAGAT, then the value of SA for that aspect of performance can 

be validated.

Framework: The framework assumes that certain knowledge is necessary to properly perform a task, 

and that we can model the set of all such elements of knowledge as the “target set,” as shown in 

Equation (1).

T T T Tn Κ ={ Κ Κ Κ 1 2 , ,, K } (1)

The “actual” set of knowledge available to the person(s), i.e., that person’s SA, is shown in Equation 

(2). (The definition of “available” is non-trivial, but is simplified here to mean “recallable”).

SA SA SA SA m Κ ={ Κ Κ Κ 1 2 , ,, K } (2)
13

A mapping function Ci f that relates a general set of knowledge Κ to some performance criterion 

relates the effect on the performance criterion of possessingΚ . For example, if one of the elements of 

SA is that my building is on fire, where the performance criterion is the probability I would evacuate, 

that probability of evacuating would be higher than if I did not possess that knowledge.

We then define subsets q

T T Κ⊂ Κ and rT T Κ⊂ Κ such that: 

 s.t. , ( ) ( ) i i ∃ Κ Cf f i CT C T > ¬ ¬ Κ ∀ Κ∩ Κ = ∅ (3)

 s.t. , ( ) ( ) i i

q qr

T CT CT ∃Κ Κ ≥ Κ ∀ ≠ f f rq (4)

where:

( ) C i i f C Κ = (5)

Equation (3) states that there must exist some performance criterion Ci such that performance given 

the knowledge from the target set is better than if no knowledge from the target set is available. 

Equation (4) states that there exists some subset of the target set that results in as good as, if not 

better, performance than any other subset.

Landry argues that unless (3) is true, situation awareness has no value as a construct. That is, if it is 

possible to achieve all performance criteria without the need to possess at least one particular and 

identifiable set of knowledge, SA is not a useful concept. It is possible that such a set is nonexclusive—there may be multiple identifiable sets, where any one of those sets is capable of 

providing the best performance; it is only necessary that these sets provide better performance than 

other sets. An important conceptual implication of this framework is that it makes the concept of SA 

falsifiable. If there are no performance criteria for which identifiable sets of knowledge improve 

performance, then SA has no validity. 

A flight simulator experiment was run in which particular aspects of performance, for which a target 

set of information and a mapping function was defined. The purpose of the experiment was, in part, 

to validate or invalidate such mapping functions. That is, this experiment was primarily designed to 

exercise the framework rather than to test the particular relationships identified. Eighteen pilot 

participants flew five profiles in which general performance data was collected, as well as their 

performance on a particular “focus event.” For each of these events, a target set of information and 

mapping function was identified. 

Half of the pilots underwent SAGAT probes during the trials, with the other half acting as a control to 

ensure that SAGAT provided neither an interruption effect nor a tip-off effect. The mapping 

functions served as a predictor of performance based on a SAGAT response to a particular question.

SAGAT produced neither a significant tip-off nor interference effect. Overall pilot performance at the 

SAGAT queries was slightly below 50%, as was performance at the focus events. The data, in 

particular the post-hoc questionnaires, have so far suggested that the mapping functions are invalid.

(The ability to invalidate a mapping function is useful in that it demonstrates the capability of the 

method to eliminate incorrect mapping functions.) Specifically, although the events were chosen for 

their simplicity of mapping, it appears that the mapping is more complex and that “inverse” mappings 

are largely inaccurate. Additional data analysis is being conducted to try to define accurate mapping 

functions which can then be verified in future experiments.
14

4.2 Distributed Air-Ground Simulations 

A series of distributed air-ground simulations were run during the period of the NRA. These involved 

simulation roles being played by two or more member organizations. Scenarios for each distributed 

simulation conducted as part of the NRA were based on sectors in Kansas City Center (ZKC) and 

Indianapolis Center (ZID), with participant pilots flying en route, arrival and approaches to Louisville 

International Airport (SDF), while avoiding weather and managing spacing, as shown in Figure 2.

Figure 2. Simulated airspace for distributed air-ground simulations.

4.2.1 Trajectory Oriented Operations with Limited Delegation during Convective 

Weather (TOOWiLDx) Simulation Demonstrations

FDDRL led two simulation demonstrations in 2008 that explored flight-crew-based arrival 

management in the context of convective weather during en route and arrival operations. We 

examined whether pilot performance was affected by cockpit weather display type (current-day radar 

vs. 3D CSD). For this project, the simulation network consisted of two consortium facilities (FDDRL 

and CSULB). Participant pilots were located at FDDRL; ATC confederates were located at CSULB 

CHAAT. This demonstration was used to design two distributed simulations in 2010.

4.2.2 Evaluation of Automated Spacing Support Tools for Interval Management 

Operations under Hazardous Weather Conditions

Planning currently underway for future airspace management assumes that flight decks will play an 

increased role in managing the intervals between ownship and a lead aircraft when approaching an 

airport. In this study, pilots were asked to achieve a specific time–in-trail while flying an arrival into 

the SDF airport. Shortly before reaching their top of the descent, pilots were responsible for avoiding 

weather. A spacing tool calculated airspeeds designed to achieve the desired time in trail at the final 

approach fix. Pilots were exposed to four experimental conditions which varied in how strictly the 

pilots were to follow these calculated speeds, and whether these speeds had to be entered into the 

autopilot manually. Giving the pilots more discretion had little effect on the final spacing interval. 

However, when pilots were required to enter speeds into the system manually, they were less accurate 

and failed to meet altitude restrictions significantly more often. Moreover, requiring the pilots to 

manually enter speeds into the system frequently led to poorer energy management and higher 

spacing interval errors at the final approach fix, even in the conditions where pilots were instructed to 

strictly follow speed guidance. This finding was traced to poorer compliance with the automated 

speed guidance, lack of awareness of this poor compliance, and insufficient awareness of the energy 

state of the aircraft. These results suggest that some form of energy guidance may be needed to 

augment interval management. To do this, recommendations were made for integrating the spacing 

interval management automation with near-term and far-term energy management systems. While 

T Merg
15

these results may not always generalize to alternative spacing implementations, one should not 

assume that pilots manually closing the loop on automated commands can perform as well as a fully 

automated system. For additional details regarding this simulation see (Dao et al., 2010; Johnson et 

al., 2010).

4.2.3 Situation Awareness with Trajectory Oriented Operations in Weather 

(SAWTOOth)

A multi-participant distributed simulation experiment was run to examine the robustness of interval 

management during continuous descent approaches (CDAs) along the CBSKT 1 Arrival into SDF.

Eight experimental pilots started the scenario in an en-route phase of flight and were asked to avoid 

convective weather while performing merging and spacing tasks along with a CDA into SDF. Two 

controllers managed the sectors through which the pilots flew, with one managing a sector that 

included the Top of Descent (ZKC-90), and the other managing a sector that included the merge point 

for arrival (ZID-91) into SDF. We determined the impact of changes in responsibility for separation 

assurance on the workload, SA, and performance of pilots and ATCs. Three plausible strategies for 

separation assurance were simulated and evaluated. 

Concept 1: Pilot Primary. Pilots flying equipped aircraft were given the primary 

responsibility for separation assurance between ownship and all other aircraft. Equipped 

aircraft had on-board conflict alerting, a route assessment tool (RAT), and auto-resolver tool.

ATC was responsible for resolving conflicts between unequipped aircraft and all conflicts 

with experimental aircraft on their CDA. The human ATC was equipped with conflict 

alerting, a trial planner, and auto-resolver tool. The ground-based auto-resolver agent had no 

responsibility.

Concept 2: ATC Primary. ATCs were given the primary responsibility for separation 

assurance. As in Concept 1, equipped aircraft had on-board conflict alerting, the RAT, and 

auto-resolver tool, but these were used for weather re-routing and proposing conflict 

resolutions. ATCs had responsibility for equipped-unequipped and unequipped-unequipped 

conflicts only, and for conflicts between any aircraft on a CDA. The autoresolver agent was 

responsible for equipped-equipped conflicts.

Concept 3: Autoresolver Primary. In Concept 3, the autoresolver agent was responsible for 

resolving most conflicts. Aircraft were not equipped with conflict alerting conflict resolution 

capabilities, but the RAT was available to aid re-routing for weather avoidance. Pilots had no 

responsibility for separation. The autoresolver agent was responsible for conflicts between 

equipped-equipped and equipped-unequipped aircraft. The human ATC was equipped with 

conflict alerting, the trial planner, and auto-resolver tool. ATC was responsible for conflicts 

between unequipped aircraft only, and for conflicts between any aircraft on a CDA.

In all scenarios, pilots initially engaged a spacing tool which was designed to achieve a spacing 

interval of 105 seconds from a designated lead aircraft at the final approach fix, and then re routed 

around weather. If a pilot felt that his/her weather avoidance maneuver made it unlikely that the 

spacing goal would be met, the pilot could request ATC to assign a new lead aircraft. Two additional 

variables were manipulated, weather complexity (high vs. low) and type of weather display (NextRad 

vs. 3D NextRad). High-complexity weather contained more weather cells, and covered more 

airspace. Low-complexity weather consisted of fewer weather cells.

A set of performance measures enabled us to determine the effects of changes in responsibility for 

traffic separation on pilots and ATCs in the experiment. However, because only two ATCs were 
16

tested in this study, the most reliable data was obtained from pilots. We evaluated the impact of each 

scenario manipulation on pilot workload, SA, spacing performance, weather avoidance, and CDA 

profile. Some of the major findings are as follows (for more information see Vu et al., 2010a,b; 

Strybel et al., 2010; Ho et al., 2010).

• The SA and workload of pilots changed with concepts of operation. Pilots indicated that all 

concepts were workable and showed little change in workload across the three concepts. Pilots 

showed higher levels of awareness when they were responsible for maintaining separation with 

other aircraft. Although the controllers also indicated that all concepts were workable, they 

reported increased workload depending on concept of operation and sector.

• The online probe method for assessing pilot SA was effective in detecting changes in pilot roles 

and responsibilities for traffic separation. The change in pilot tasks regarding traffic separation 

could be determined from the response latencies for questions related to traffic conflicts. These 

changes in awareness could not be attributed to workload because workload probe latencies 

were unaffected by changes in responsibility.

• Weather perturbations can create situations in which aircraft spacing can be negatively affected 

regardless of whether the pilot faithfully followed the leading aircraft around weather. If pilots 

do not fully understand the rationale and logic of the spacing automation and its limitations, 

then trusting the automation to be able to resolve a spacing error (i.e., being ahead or behind 

schedule) to regain a lost or interrupted spacing assignment requires that pilots be highly

confident in the automation. 

5. Conclusions

Our multidisciplinary, research consortium of scientists and engineers from California State University 

Long Beach, San Jose State University Foundation, California State University Northridge, Purdue 

University, and The Boeing Company was successful in meeting the majority of our proposed NRA 

objectives. Because of the knowledge and experience of our members, as well as the unique approaches 

taken by them, we were able to work in parallel to identify SA and performance requirements brought 

about by anticipated changes in NextGen airspaces, develop SA and workload metrics, distributed 

simulation capabilities for evaluating NextGen concepts of operation, and assess the impacts of 

potential NextGen concepts of operations on SA, workload, and performance.

As a result of this NRA, we have gained a greater understanding of SA and its measurement, and have 

shared our knowledge with the scientific community: During the NRA, we published 20 technical 

papers, made 21 presentations at conventions, presented 6 briefings to NASA, and prepared 5 

unpublished reports. Additional technical papers based on the NRA work were also published after the 

funding period. Our metric work improved and validated existing tools for measuring SA and 

workload, and the resulting probe tool we developed can be used by air traffic management scientists 

and engineers for assessing new concepts of operation, determining the effectiveness of automated 

tools, comparing different interface designs, and establishing training requirements. We also 

recommended a set of performance measures that will make it easier for researchers to compare results 

between experiments, simulations, and operational settings. 

Through the success of our distributed-simulation network, we have shown how to create a flexible 

and cost-effective test bed for simulations and research involving current and future airspaces. This 

network provides a mechanism for consortium members, colleagues, and students to pursue research 

on other topics in air traffic management and aviation, thus enabling them to make greater 

contributions to the field. 
17

6. Publications and Presentations Citing the NRA

Year 1 (October 2006–September 2007)

Publications

Battiste, V., Johnson, W. W., Johnson, N. H., Granada, S., & Dao, Q. (2007). Flight crew perspective 

on the display of 4D information for en route and arrival merging and spacing. In J. Jacko (Ed), 

Human-Computer Interaction, Part I, HCII 2007, Lecture Notes in Computer Science 4551

ISBN: 978-3-540-73106-1. Berlin: Springer-Verlag.

Strybel, T. Z., Vu, K.-P. L., Dwyer, J. P., Kraft, J., Ngo, T. K., Chambers, V., & Garcia, F. P. (2007).

Predicting perceived situation awareness of low altitude aircraft in terminal airspace using probe 

questions. In J. Jacko (Ed), Human-Computer Interaction, Part I, HCII 2007, Lecture Notes in 

Computer Science 4550 ISBN: 978-3-540-73104-7. Berlin: Springer-Verlag.

Presentations

Battiste, V., Johnson, W. W., Johnson, N. H., Granada, S., & Dao, Q. (2007, July). Flight crew 

perspective on the display of 4D information for en route and arrival merging and spacing. Paper 

presented at Human Computer Interaction International 2007 Conference. Beijing, China.

Strybel,T. Z., Vu, K.-P. L., Dwyer, J. P., Kraft, J., Ngo, T. K., Chambers, V., & Garcia, F. P. (2007, 

July). Predicting perceived situation awareness of low altitude aircraft in terminal airspace using

probe questions. Paper presented at Human Computer Interaction International 2007 Conference. 

Beijing, China.

Strybel, T. Z., Vu, K.-P. L., Battiste, V., & Johnson, W. W. (2007, November). Conducting 

Distributed Simulations Over the Internet. Poster accepted for presentation at the 37th Annual 

Meeting of the Society for Computers in Psychology. Long Beach, CA.

Unpublished Reports

Strybel, T. Z., Vu, K.-P. L., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Landry S. (2007). NRA 

Year 1 Report.

Briefings

Strybel et al. (2006). Metrics for Operator Situation Awareness, Workload and Performance in 

Automated Separation Assurance Systems Kickoff Briefing, January, 2007.

Strybel et al., (2007). Metrics for Operator Situation Awareness, Workload and Performance in 

Automated Separation Assurance Systems. Presented at the NASA Ames Technical Interchange 

Meeting, March, 2007
18

Year 2 (October 2007–September 2008)

Publications

Pierce, R., Strybel, T., & Vu, K.-P. L. (2008). Comparing situation awareness measurement 

techniques in a low fidelty air traffic control simuluation. Proceedings of the 26th International 

Congress of the Aeronautical Sciences (ICAS). Anchorage, AS.

Pierce, R., Vu, K.-P. L., Nguyen, J., & Strybel, T. (2008). The relationship between SPAM, 

workload, and task performance on a simulated ATC task. Proceedings of the 52nd Human 

Factors and Ergonomics Society Annual Meeting. New York, NY.

Strybel, T. Z., Vu, K.-P. L., Kraft, J., & Minakata, K. (2008). Assessing the situation awareness of 

pilots engaged in self spacing. Proceedings of the 52nd Human Factors and Ergonomics Society

Annual Meeting. New York, NY.

Battiste, V., Johnson, W. W., Dao, A.-Q., Brandt, S., Johnson, N., & Granada, S. (2008).

Assessment of Flight Crew Acceptance of Automated Resolutions Suggestions and Manual 

Resolution Tools. Proceedings of the 26th International Congress of the Aeronautical Sciences 

(ICAS). Anchorage, AS.

Presentations 

Pierce, R., Strybel, T., & Vu, K.-P. L. (2008). Comparing situation awareness measurement 

techniques in a low fidelty air traffic control simuluation. Presented at the 26th International 

Congress of the Aeronautical Sciences (ICAS). Anchorage, AS.

Pierce, R., Vu, K.-P. L., Nguyen, J., & Strybel, T. (2008). The relationship between SPAM, 

workload, and task performance on a simulated ATC task. Presented at the 52nd Human Factors 

and Ergonomics Society Annual Meeting. New York, NY.

Strybel, T. Z., Vu, K.-P. L., Kraft, J., & Minakata, K. (2008). Assessing the situation awareness of 

pilots engaged in self spacing. Presented at the 52nd Human Factors and Ergonomics Society

Annual Meeting. New York, NY.

Battiste, V., Johnson, W. W., Dao, A.-Q., Brandt, S., Johnson, N., & Granada, S. (2008). Assessment of 

Flight Crew Acceptance of Automated Resolutions Suggestions and Manual Resolution Tools.

Presented at the 26th International Congress of the Aeronautical Sciences (ICAS). Anchorage, AS.

Martin, P., Ho, N. and Berson, B. (2008) Identification of shared situation awareness information 

requirements for NGATS concepts. Presented at Human Factors and NextGen; The Future of 

Aviation, Arlington TX, May 2008.

Unpublished Reports

Dao, A-Q & Strybel, T.Z. (2008). Demonstration of distributed simulation capabilities. Unpublished 

Report Prepared for the Consortium for the Study of Situation Awareness.

Dwyer, J.D. & Landry, S. (2008). Roles and responsibilities conceptualizations for next generation 

air transportation system. Unpublished Report Prepared for the Consortium for the Study of 

Situation Awareness.

Strybel, T. Z., Vu, K.-P. L., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Landry S. (2008). NRA 

Year 2 Report.
19

Briefings

Strybel et al. Metrics for operator situation awareness, workload and performance in automated 

separation assurance systems: year 2 progress report. Presented to Todd Farley and FDDRL, 

October, 2008.

Year 3 (October 2008–September 2009)

Publications

Dao, A. Q., Brandt, S. L., Battiste, V., Vu, K.-P. L., Strybel, T. Z., & Johnson, W. W. (2009). The 

impact of automation assisted aircraft separation on situation awareness. Human Interface, Part 

II, HCII 2009, Lecture Notes in Computer Science, 5618, 738-747.

Dwyer, J. P., & Landry, S. (2009). Separation assurance and collision avoidance concepts for the 

Next Generation Air Transportation System. Human Interface, Part II, HCII 2009, Lecture Notes 

in Computer Science, 5618, 748-757.

Ho, N. T., Martin, P., Bellissimo, J., Berson, B. (2009). Information requirements and sharing for 

NextGen function allocation concepts. Human Interface, Part II, HCII 2009, Lecture Notes in 

Computer Science, 5618, 776-785.

Ligda, S. V, Johnson, N., Lachter, J., & Johnson, W. W. (2009). Pilot confidence with ATC 

Automation using cockpit situation display tool in a distributed traffic management environment.

Human Interface, Part II, HCII 2009, Lecture Notes in Computer Science, 5618, 758-775.

Strybel, T., Minakata, K., Nguyen, K, Pierce, R., & Vu, K.-P. L. (2009). Optimizing online situation 

awareness probes in air traffic management tasks. Human Interface, Part II, HCII 2009, Lecture 

Notes in Computer Science, 5618, 845-854.

Vu, K.-P. L., Minakata, K., Nguyen, J, Kraut, J., & Raza, H. (2009). Situation awareness and 

performance of students versus experienced air traffic controllers. Human Interface, Part II, HCII 

2009, Lecture Notes in Computer Science, 5618, 865-874.

Presentations

Dao, A. Q., Brandt, S. L., Battiste, V., Vu, K.-P. L., Strybel, T. Z., & Johnson, W. W. (2009). The 

impact of automation assisted aircraft separation on situation awareness. Presented at HCI 

International, San Diego, CA, July, 2009.

Dwyer, J. P., & Landry, S. (2009). Separation assurance and collision avoidance concepts for the 

Next Generation Air Transportation System. Presented at HCI International, San Diego, CA, 

July, 2009.

Ho, N. T., Martin, P., Bellissimo, J., Berson, B. (2009). Information requirements and sharing for 

NGATS function allocation concepts. Presented at HCI International, San Diego, CA, July, 2009.

Ligda, S. V, Johnson, N., Lachter, J., & Johnson, W. W. (2009). Pilot confidence with ATC 

automation using cockpit situation display tool in a distributed traffic management environment.

Presented at HCI International, San Diego, CA, July, 2009.

Strybel, T., Minakata, K., Nguyen, K, Pierce, R., & Vu, K.-P. L. (2009). Optimizing online situation 

awareness probes in air traffic management tasks. Presented at HCI International, San Diego, 

CA, July, 2009.

Vu, K.-P. L., Minakata, K., Nguyen, J, Kraut, J., & Raza, H. (2009). Situation awareness and 

performance of students versus experienced air traffic controllers. Presented at HCI 

International, San Diego, CA, July, 2009.
20

Unpublished Reports

Strybel, T. Z., Vu, K.-P. L., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Landry S. (2007). NRA 

Year 3 Report.

Briefings

Strybel, T.Z. & Johnson, W. Situation awareness with trajectory oriented operations in weather

(SAWTOOth): Design Review. Presented to NASA Ames March, 2009.

Year 4 (October 2010–January 2011)

Publications

Dao, A-Q.V., Lachter, J., Battiste, V., Brandt, S.L., Vu, K-P.L., Strybel, T.Z., Ho, N., Martin, P., & 

Johnson, W.W. (2010). Automated spacing support tools for interval management operations 

during continuous descent approaches. In Proceedings of the 54th Annual Meeting of the Human 

Factors & Ergonomics Society, San Francisco, CA: Human Factors Society.

Ho, N., Johnson, W., Martin, P., Lachter, J., Dao, A.-Q., Brandt, S & Battiste, V. (2010). Pilot 

response to off-nominal conditions in merging and spacing operation. Proceedings of the 29th 

Digital Avionics Systems Conference. Salt Lake City, Utah: IEEE.

Johnson, W., Ho, N., Battiste, V., Vu, K.P. L., Lachter, J., Ligda, S., Dao, A.-Q., & Martin, P. (2010).

Management of continuous descent approach during interval management operation. Proceedings 

of the 29th Digital Avionics Systems Conference. Salt Lake City, Utah: IEEE.

Ligda, S.V., Dao, A-Q.V., Strybel, T.Z., Vu, K-P.L., Battiste, V., & Johnson, W.W. (2010). Impact of 

conflict avoidance responsibility allocation on pilot workload in a distributed air traffic 

management system. In Proceedings of the 54th Annual Meeting of the Human Factors & 

Ergonomics Society, San Francisco, CA: Human Factors Society.

Strybel, T.Z., Vu, K-P.L., Bacon, L.P., Kraut, J., Battiste, V., & Johnson, W. (2010). Diagnosticity of 

an online query technique for measuring pilot situation awareness in NextGen. In Proceedings of 

the 29th Digital Avionics Systems Conference, Salt Lake City, UT. 

Vu, K-P.L., Strybel, T.Z., Kraut, J., Bacon, L.P., Minakata, K., Nguyen, J., Rotterman, A., Battiste, 

V., & Johnson, W. (2010). Pilot and controller workload and situation awareness with three 

traffic management concepts. In Proceedings of the 29th Digital Avionics Systems Conference, 

Salt Lake City, UT. 

Vu, K-P. L., Strybel, T. Z., Battiste, V. & Johnson, W. (in press). Factors Influencing the Decisions 

and Actions of Pilots and Controllers in Three Plausible NextGen Environments. In R. W. Proctor 

and S. Nof. Y. Yih, (Eds.) Cultural Factors in Decision Making and Action. CRC Press.

Presentations

Dao, A-Q.V., Lachter, J., Battiste, V., Brandt, S.L., Vu, K-P.L., Strybel, T.Z., Ho, N., Martin, P., & 

Johnson, W.W. (2010). Automated spacing support tools for interval management operations 

during continuous descent approaches. Presented at the 54th Annual Meeting of the Human 

Factors & Ergonomics Society, September 2010, San Francisco, CA.

Ho, N., Johnson, W., Martin, P., Lachter, J., Dao, A.-Q., Brandt, S & Battiste, V. (2010). Pilot 

response to off-nominal conditions in merging and spacing operation. Presented at the 29th 

Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah. 
21

Johnson, W., Ho, N., Battiste, V., Vu, K.P. L., Lachter, J., Ligda, S., Dao, A.-Q., & Martin, P. (2010).

Management of continuous descent approach during interval management operation. Presented at 

the 29th Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah.

Ligda, S.V., Dao, A-Q.V., Strybel, T.Z., Vu, K-P.L., Battiste, V., & Johnson, W.W. (2010). Impact of 

conflict avoidance responsibility allocation on pilot workload in a distributed air traffic 

management system. Presented at the 54th Annual Meeting of the Human Factors & Ergonomics 

Society, September 2010, San Francisco, CA.

Strybel, T.Z., Vu, K-P.L., Bacon, L.P., Kraut, J., Battiste, V., & Johnson, W. (2010). Diagnosticity of 

an online query technique for measuring pilot situation awareness in NextGen. Presented at the 

29th Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah,

Vu, K-P.L., Strybel, T.Z., Kraut, J., Bacon, L.P., Minakata, K., Nguyen, J., Rotterman, A., Battiste, 

V., & Johnson, W. (2010). Pilot and controller workload and situation awareness with three 

traffic management concepts. Presented at the 29th Digital Avionics Systems Conference, 

October, 2010, Salt Lake City, Utah,

Vu, K-P. L., Strybel, T. Z., Battiste, V. & Johnson, W. (2010). Factors Influencing the Decisions and 

Actions of Pilots and Controllers in Three Plausible NextGen Environments. Presented at 

Cultural Factors in Decision Making and Action, November 2010, Lafayette, Indiana.

Unpublished Reports

Dwyer, J. P., Gershzohn, G., & Thorpe, E.M. Review of Air Traffic Controller and Pilot Training for 

The Sawtooth II Airborne Self-Separation, Continuous Descent Arrival NextGen Air Traffic 

Management Concept Simulation. Presented to NRA Consortium January 2011.

Strybel, T.Z., Vu, K.P.L., Chiappe, D. Battiste, V. & Dao, A-Q. V. Metrics for Situation Awareness, 

Workload, and Performance: Manual for Online Probe Administration. Presented to Walt 

Johnson NRA Technical Monitor January, 2011.

Post-Award Period Publications Citing Work Supported by the NRA

Bacon, L.P., Strybel, T.Z. Vu, K.-P.L., Kraut, J.M., Nguyen, J.H., Battiste, V., & Johnson, W. W. 

(2011) Situation awareness, workload and performance in midterm NextGen: Effect of variations

in aircraft equipage levels between scenarios. Proceedings of the 2011 International Symposium

on Aviation Psychology (pp. 32-37), Dayton, OH.

Kiken, A., Rorie, R.C, Bacon, L. P., Billinghurst, S., Kraut, J.M., Strybel, T. Z., Vu, K.-P. L., &

Battiste, V. (2011). Effect of ATC training with NextGen tools and online situation awareness 

and workload probes on operator performance. In G Salvendy & M. J. Smith (Eds.), Human 

Interface, Part II, HCII 2011, Lecture Notes in Computer Science, 6772, 483-492.

Kraut, J., Kiken, A., Billinghurst, S., Morgan, C., Strybel, T. Z., Chiappe, D. L., & Vu, K.-P. L. 

(2011) Effects of data communications failure on air traffic controller sector management 

effectiveness, situation awareness and workload. In G. Salvendy and M. J. Smith (Eds.), Human 

Interface, Part II, HCII 2011, Lecture Notes in Computer Science, 6772, 493-499.

Landry, S., J., & Surakitbanharn, C. (2011). A formalism for assessing the situation awareness of 

pilots. In G. Salvendy and M. J. Smith (Ed.s), Human Interface, Part II, HCII 2011, Lecture 

Notes in Computer Science, 6781, 572-581.
22

Strybel, T. Z., Vu, K. L., Bacon, L. P., Kiken, A., Billinghurst, S., Rorie, R. C., Morgan, C., &,

Battiste, V. (2011) Situation awareness, workload and performance in midterm NextGen: Effect of 

dynamic variations in aircraft equipage levels. Proceedings of the 2011 International Symposium

on Aviation Psychology, (pp 118-123). Dayton, OH.

Strybel, T. Z., Vu, K.-P. L, Chiappe, D., Battiste, V., Johnson, W., & Dao, Q., V. (2011). Metrics for 

situations awareness, workload, and performance: Manual for online probe administration.

(Report no. NASA/TM-2011-215991). Moffett Field, CA: NASA Ames Research Center.

7. References

Chiappe D. L., Strybel,T. Z., & Vu, K.-P.L. (in press). Mechanisms for the acquisition of situations 

awareness in situated agents. Theoretical Issues in Ergonomics Sciences, DOI: 

10.1080/1463922X.2011.611267

Durso, F. T., & Dattel, A. (2004). SPAM: The real-time assessment of SA. In S. Banbury & S. 

Trembley (Eds.), A cognitive approach to situation awareness: Theory, measures and application

(pp. 137-154). New York: Aldershot.

Endsley, M. R. (1990). Measurement of situation awareness in dynamic systems. Human Factors, 37, 

65-84.

Endsley, M. R., & Smolensky, M. (1998). Situation awareness in air traffic control: The Picture. In 

M. Smolensky & E. Stein (Eds.) Human factors in air traffic control (pp. 115-154). New York: 

Academic Press.
23
Report Documentation Page Form Approved

OMB No. 0704-0188

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YY)

12-15-11

2. REPORT TYPE

Technical Memorandum

3. DATES COVERED (From – To)

4. TITLE AND SUBTITLE

Metrics for Operator Situation Awareness, Workload, and 

Performance in Automated Separation Assurance Systems

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Thomas Z. Strybel, Kim-Phuong L. Vu, Vernol Battiste, ArikQuang Dao, John P. Dwyer, Steven Landry, Walter Johnson, 

Nhut Ho

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

411931.02.51.01.08

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESSES(ES)

NASA Ames Research Center

Moffett Field, California 94035-1000

8. PERFORMING ORGANIZATION 

REPORT NUMBER

TH-091

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

10. SPONSORING/MONITORʼS ACRONYM(S)

NASA

1.1.1.1.1 11. SPONSORING/MONITORING

REPORT NUMBER

NASA/TM–2011-215997

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified—Unlimited

Subject Category: 3 Availability: NASA CASI (301) 621-0390 Distribution: Nonstandard

13. SUPPLEMENTARY NOTES

Point of Contact: Walter Johnson, NASA Ames Research Center, MS 262-4, Moffett Field, CA 94037; 

(650) 604-3667.

14. ABSTRACT

A research consortium of scientists and engineers from California State Univ. Long Beach, San Jose State Univ. 

Foundation, California State Univ. Northridge, Purdue Univ., and the Boeing Company was assembled to evaluate the 

impact of changes in roles and responsibilities and new automated technologies, being introduced in the Next Generation 

Air Transportation System, on operator situation awareness and workload. To meet these goals, consortium members 

performed systems analyses of NextGen concepts and airspace scenarios and concurrently evaluated SA, workload, and 

performance measures to assess their appropriateness for evaluations of NextGen concepts and tools. The following 

activities and accomplishments were supported by the NRA: a distributed simulation, metric development, systems 

analysis, part-task simulations, and large-scale simulations. As a result of this NRA, we have gained a greater 

understanding of situation awareness and its measurement, and have shared our knowledge with the scientific 

community. This network provides a mechanism for consortium members, colleagues, and students to pursue research 

on other topics in air traffic management and aviation, thus enabling them to make greater contributions to the field. 

15. SUBJECT TERMS

situation awareness; workload; NextGen

16. SECURITY CLASSIFICATION OF: 17. LIMITATION 

OF 

ABSTRACT

UU

18. NUMBER 

OF 

PAGES

32

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk at email: 

help@sti.nasa.gov a. REPORT

U

b. ABSTRACT

U

c. THIS PAGE

U

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

Standard Form 298 (Rev. 8-98)