ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)[EN-034] A Controller-in-the-Loop Simulation of Ground-Based
Automated Separation Assurance in a NextGen Environment(EIWAC 2010)+

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall

San Jose State University | NASA Ames Research Center
Human-Systems Integration Division
Moffett Field, CA., USA[jeffrey.r.homola | thomas.prevot | joey.mercer | connie.l.brasil | lynne.h.martin | christopher.d.cabrall]@nasa.gov
Abstract: A controller-in-the-loop simulation was conducted in the Airspace Operations Laboratory (AOL) at the NASA
Ames Research Center to investigate the functional allocation aspects associated with ground-based automated separation
assurance in a far-term NextGen environment. In this concept, ground-based automation handled the detection andresolution of strategic and tactical conflicts and alerted the controller to deferred situations. The controller was responsible
for monitoring the automation and managing situations by exception. This was done in conditions both with and without
arrival time constraints across two levels of traffic density. Results showed that although workload increased with anincrease in traffic density, it was still manageable in most situations. The number of conflicts increased similarly with arelated increase in the issuance of resolution clearances. Although over 99% of conflicts were resolved, operational errors
did occur but were tied to local sector complexities. Feedback from the participants revealed that they thought they
maintained reasonable situation awareness in this environment, felt that operations were highly acceptable at the lower
traffic density level but were less so as it increased, and felt overall that the concept as it was introduced here was a positive
step forward to accommodating the more complex environment envisioned as part of NextGen.
Keywords: controller-in-the-loop, AOL, NASA, NextGen, ground-based automation, separation assurance

1. INTRODUCTION
In the United States’ National Airspace System (NAS) of
today, air traffic controllers manage operations within theirsector of responsibility through active involvement witheach aircraft that enters and exits. This allows thecontroller to incorporate each aircraft into their overall
operational picture and has facilitated the current levels ofsafety. However, the airspace envisioned for the Next
Generation Air Transportation System (NextGen) is
expected to be both higher in traffic density andcomplexity relative to what is experienced today (Fig. 1
presents an example of how such an environment would be
displayed without changes to the current system) [1]. The
transition to this future system will certainly requirechanges in a number of areas, without which operations
within the NAS would likely become unviable.

Figure 1 Current day controller display with traffic density reachinglevels nearly threefold what is experienced today

Given that the primary responsibility of air traffic
controllers is to maintain safe separation between aircraft,the area of separation assurance (SA) has been identified
as one requiring a concerted and focused research effort
with advances in this domain serving as enablers for thetransition to NextGen. Within this framework, the thrust ofsome of the recent research has been on addressing theimpact of higher levels of traffic density and complexity on

the workload of air traffic controllers –a potential barrier
to NextGen realization- and exploring the proper levels of
automated support necessary for the accommodation ofincreased traffic and complexity without creating excessive
workload and compromising safety.
This paper will first outline the concept of operations thatsome of the research has been predicated upon followed by213

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrallbrief descriptions of previous SA research conducted in the
Airspace Operations Laboratory (AOL) [2] at the NASA
Ames Research Center. This will then be followed by thepresentation of the most recent research effort withaccompanying results and will conclude with a discussionof those results.

concept as well as the environment in which it would be
realized were limited.
To date there have been three controller-in-the-loop
simulations conducted in the AOL specifically related toseparation assurance. Each of these has built upon the
research and results of the previous study and has served to
inform the next while incrementally introducing greaterlevels of automation. For simplicity these studies arereferred to as SA1, SA2, and SA3 according to theirtemporal order.

2. GROUND-BASED AUTOMATED SEPARATIONASSURANCE
In today’s environment, separation assurance is maintained
through a largely manual process where the air traffic
controller scans and identifies potential separation risks,develops a conflict avoidance resolution based on theirunderstanding of the current operational picture, and issues
a verbal clearance to the appropriate aircraft. While thisprocess has been and is carried out very effectively, it will
become an increasingly difficult task if and when traffic
demand in the United States reaches the increased levels
forecast by the Federal Aviation Administration [3].

3.1 SA1
The first in this series, referred to as SA1, was conducted
in 2007 and evaluated the differences in performance
between participants resolving conflicts in a manual andinteractive mode as well as a fully automated mode [6-7].
This was done at current day (1x), twice (2x), and threetimes (3x) that level of traffic. At the heart of theinteractive and fully automated modes lay the autoresolver
algorithm outlined as part of the Advanced Airspace
Concept. Workload impact and acceptability of thealgorithm’s resolutions were also investigated. Results
suggested that the automation provided significant benefits
in terms of safety and efficiency particularly at higher
levels of traffic. There was also a significant reduction inworkload. The resolutions provided by the automation
were also rated as being generally acceptable.

One of the changes that would provide for the ability toaccommodate such levels of traffic is the introduction ofgreater levels of automation in support of the separation
assurance function. Assuming an environment wheretrajectory based operations (TBO) are being conducted,
such automation would perform the duties of conflict
detection and develop resolutions for nominal conflictsituations and uplink conflict-free trajectories to theappropriate aircraft via data communication channels.
Tactical conflict situations with an imminent loss of
separation would be handled by an additional layer of
automation where tactical vectors and/or altitude changes
would be sent to the appropriate aircraft via a separate
communications channel dedicated to such transmissions.

3.2 SA2
The second study, SA2, was conducted in 2008 and tested
air/ground operations in an environment where groundbased automation was responsible for safely managing
aircraft trajectories [8]. Controller participants were
responsible for handling pilot requests that were deferredby the automation and also handled scripted off-nominal
events and tactical conflict situations both with and without
Tactical Separation Assurance Flight Environment(TSAFE) support [9]. Operations were conducted at 2x
and 3x levels of traffic. Results showed that the strategic
conflict resolution automation was able to resolve 98% ofconflicts, 95% of uplinked trajectories were rated as
acceptable to the flight crew participants, and workload
was generally low. Of the tactical conflicts, 75% wereresolved, which served as a springboard for discussion of
the issues related to such automation and how they could
be addressed in future research [10].

Such changes to the task of separation assurance and the
involvement of such levels of automation would signal a
shift away from an individual air traffic controller having
the sole responsibility for maintaining safe separation to
one in which the responsibility rests with the service
provider- a term used to refer to the collaborative teamcomposed of the air traffic controller and supporting
automation. This shift would not only allow for increased
airspace capacity, but would also allow for the air traffic
controller to provide additional services while managing
the airspace by exception.
3. PREVIOUS SA RESEARCH IN THE AOL
The components of the larger concept of ground-based
automated separation assurance as well as the concept
itself have previously been detailed and presented as partof the Advanced Airspace Concept (AAC) [4-5]. Early
simulations and analyses in this particular area wereconducted through fast-time simulations and modeling, butinvestigations into the operational and human factors
issues surrounding such changes as envisioned in the

3.3 Current Study (SA3)
The third study in this series, SA3, was conducted in partto begin addressing the concerns expressed by the Joint
Planning and Development Office (JPDO) regarding the“lack of clarity” surrounding the functional allocation of
new functions and responsibilities between the groundbased air traffic control systems and the flight deck- based
systems [11]. To that end, two separate but collaborative
studies were conducted with respect to the functional

2

214

ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)

allocation of separation assurance with schedulingconstraints between the ground and the air, with theground-based focus residing at the NASA Ames ResearchCenter and the flight deck-based focus residing at theNASA Langley Research Center. Results and discussionsof the collaborative aspect of this overall study have been
published previously [12]. This paper will focus
specifically on the ground-based portion of the study witha further examination of the allocation of separation
assurance functions between the ground-based automationand the human air traffic controller.

Figure 3 Design matrix for the medium duration runs

4.2 Operational Environment
For this study, the operational environment was alignedwith what is envisioned for NextGen where there is an
integration of the flight environment and avionics, groundbased automation, and the controller workstation.

4. METHOD
This study was conducted over the course of eight days in
February and March of 2010 in the AOL at the NASA
Ames Research Center. Training and familiarization was
performed for the first three days with the remaining fivedays being devoted to data collection.

Flight Environment
The flight environment assumed full air-ground data
communication and Automatic Dependent SurveillanceBroadcast (ADS-B) equipage for all aircraft transiting theairspace. Trajectory Based Operations were being
conducted where it was assumed that all flights were on
their user-preferred, 4-D trajectories and were cleared fortheir departure and arrival transitioning phases of flight.

This section will begin with an overview of the experiment
design and operational environment as it was presented in
the study, followed by a description of the apparatus,
participants, airspace, and the procedures used during thedata collection phase of the study.

Ground-Based Automation
The instantiation of ground-based automated separation
assurance for this study involved a number of related
components working in sequence. The first provided thedetection and reporting of conflicts with a 12 minute lookahead time, followed by the development of conflict
resolutions that were calculated to minimize delay, andfinally the uplinking of trajectories to the appropriate
aircraft without the need for controller involvement innominal situations. Successful completion of this sequence
relied upon accurate speed and position information.
However, trajectory uncertainties and inaccuracies did
exist as they do today and most likely will in the future.
Conformance monitoring of flight trajectories detected
when an aircraft had deviated from its path outside of
acceptable tolerances, at which point the particular flight
was alerted to the controller through an enlarged target
symbol and a tactical, dead reckoning conflict probe wasactivated with a five minute look ahead. In cases where aconflict was detected late, i.e., with less than three minutes
to loss of separation, a tactical vector was computed forone or both aircraft in the conflict pair and uplinked when
the time to loss of separation fell below two minutes. This
resulted in aircraft receiving the uplink to vector according
to the uplinked trajectory, which resulted in the aircraft
becoming off-track. It was then the controllers’
responsibility to put the off-track aircraft back onto theirpreferred trajectory in a safe manner.

4.1 Design
The overall ground-based study included three different
components. The first consisted of short length (S), 15minute runs that involved three different types ofscheduling constraints (see Fig. 2). The second consistedof medium length (M), 30-minute runs that involved
scheduling or no scheduling across two different traffic
densities referred to as NextGen Levels A and B (see Fig.
3). NextGen Level A was roughly equivalent to 1.5 times
current day traffic levels and NextGen Level B wasroughly equivalent to 2.0 times. The S and M runs wereused to satisfy the collaborative requirements of the largerfunctional allocation study. The third component of thestudy was an independent aspect used in a more
exploratory context that involved long, three hour runs
with scheduling and weather constraints and traffic density
levels that fluctuated to the higher extremes. The
procedures and results presented in this paper will focus
mainly on the medium length runs.

Figure 2 Design matrix for the short duration runs

Controller Workstation
The workstations and displays used for the sector
controllers and supervisors incorporated advances in thedisplay of information and access to functionality deemednecessary for control operations in a NextGen environment.

215

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall

Fig. 4 presents a screenshot of an R-side’s display as it was
implemented in the study where it can be seen that it
differed significantly from what one would find today andas shown in Fig. 1.

well the scheduling was being adhered to. In the event thatan aircraft began falling out of conformance, the timeline
provided an interactive function that allowed the controller
to call upon an automated algorithm that would attempt tofind the optimal, conflict-free solution through speeds
and/or vectors to meet the scheduling constraint.
Another item in the display that assisted in the monitoringand management task of the controller was the conflict list
seen near the upper right hand corner of Fig. 4. While theground-based automation handled the nominal conflict
detection and resolution tasks independent of the controller
in this study, conflicts relevant to the sector were displayedin this list with additional information regarding thevertical profiles of the involved aircraft, predicted time toloss of separation, closest point of approach, as well as theautomation’s status and progress in resolving the conflict.
Constraints on the automation were put in place thatprevented heading changes of more than 31 degrees oraltitude changes of more than 670 m (2200 ft) from being
sent directly to aircraft without controller involvement. In
such cases, the resolution of the conflict was alerted to thecontroller through the conflict list, at which point theywould develop a resolution either on their own or byreactivating the conflict resolution algorithm.
In both the scheduling and conflict resolution cases, thecontroller had access through the workstation and display
to data communication capabilities. Once an acceptable
solution to a given situation was arrived at, the controller
had the option to type an uplink command into thecomputer readout device (CRD) or access a fly-out menu
either through the aircraft’s data block or the callsigns in
the respective lists where the uplink command could be
activated. This then sent the modified trajectory directly tothe aircraft. Similar to this functionality was one that wasused for cases that required coordination rather than directcontact. In these cases, rather than selecting the uplinkcommand, the controller selected the coordinate clearance
command. This sent a proposed trajectory to the sector that
had control of the relevant aircraft for their approval. If the
proposal was acceptable, the receiving controller could
then uplink the clearance directly to the aircraft.

Figure 4 Advanced controller display at high levels of traffic density
where scheduling and weather avoidance tasks were being performed

The first difference one may notice in Fig. 4 is that thetarget symbols were chevrons with an associated limited
data block, which displayed the current altitude. This is anecessary departure from today’s display due to the
problem of data block clutter and overlap once aircraft
counts in a sector reach levels predicted for NextGen. In
this example, there are nearly 50 aircraft in the sector. This
would result in a controller using today’s display to spend
most, if not all, of their time simply managing data block
positions. What enables this reduction in displayed
information is the shift described earlier away from active
control of each aircraft as it is done today, to one in which
the automation manages the traffic and separation
functions while the controller monitors and manages theautomation. In this case, the information available througha data block is accessed on an as-needed basis rather thanhaving the requirement of being displayed constantly while
the aircraft is within the sector’s boundaries or under thesector’s control.

4.3 Apparatus
The operational environment just described was simulatedusing the Multi Aircraft Control System (MACS) software
package [13]. This is a Java based simulation platform
developed and maintained by the AOL software team.
MACS is a scalable platform used for the prototyping ofdisplays and concepts that range from current day to the
exploratory far term timeframe.

Additional information and interactive functionality were
integrated into the workstation to assist in the performanceand monitoring of certain tasks. For example, on the righthand side of Fig. 4 one can see a timeline displayed that
controllers used to manage the scheduling constraints ineffect for area airports. This timeline featured thescheduled time of arrival on one side and the estimatedtime of arrival for aircraft scheduled to a particular airport
or fix on the other. This served as an indication of how

MACS was the common thread shared between all of thestations and positions used throughout the simulation. This
included radar controller workstations (see Fig. 5) whichwere composed of standard PCs with 75cm Barco monitors

216

ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)

and Display System Replacement (DSR) keyboards andtrackballs as input devices. These workstations were also
equipped with tablet PCs which were used for voicecommunications similar to the Voice Communications
System (VCS) used currently. Area supervisors had access
to two separate stations, each one with a different purpose.
One station projected a Traffic Situation Display of the test
airspace with accompanying traffic load statuses andpredictions for the test sectors. The other station was
similar to the radar controller station but was configured
differently to enable the performance of strategic planning
functions. Four confederate “Ghost” controller stations
were used for the control of air traffic outside of the test
airspace, which consisted of standard PC setups with 76cm
monitors. Ten pseudopilot stations with standard PC setups
were also used for the management of individual flights inthe problems. Two simulation manager stations were also
used for the initiation and termination of each training anddata collection run.

enroute airspace with the floors of the sectors set at flightlevel (FL) 290. For the medium length runs, four test
sectors were used: ZKC 90, ZKC98, ZID81, and ZID80.
As seen in Fig. 6, the design of each sector is slightly
different and the traffic flows and characteristics produced
differences in density and complexity between them.
Arrivals and departures from local area airports
contributed to this complexity for some of the sectors withtransitioning aircraft interacting with the level flights
transiting the airspace. Confederate controllers handled thetraffic outside of the four test sectors as well as the aircraft
below FL290.

Figure 6 High altitude test airspace from the central United States usedduring the study

4.6 Procedure
The first three days of the simulation were dedicated totraining the participants on the tools, airspace, andconcepts that they would be presented during the datacollection phase. This was done through a combination ofbriefing discussions, overviews and presentations in thelaboratory, and hands-on interaction with the workstations
and software. The hands-on portion of the training was
conducted by using traffic scenarios of varying levels of
traffic density ranging from very light to the full scaledensities that they would experience during the formal
simulation.

Figure 5 Radar controller workstations with TSD and test sector load
predictions projected onto the wall above

4.4 Participants
Six current FAA front line managers, each from different
enroute centers and current on radar, served as test
participants for this study. Each rotated through radar andarea supervisor positions in order to operate and evaluate
the concept from the two different perspectives. Four
recently retired controllers worked the remaining test
positions. Four additional retired controllers acted asconfederate “Ghost” controllers that managed the air
traffic outside of the test area. Ten general aviation pilots
acted as pseudopilots and were assigned to each of the test
sectors and the surrounding areas.

The traffic scenarios used in the short and medium runs
were initially developed by the team in the ATOL as were
the formulations of the STA assignments that would be
issued to aircraft in the appropriate runs. These traffic
scenarios were then collaboratively adjusted and modified
to ensure proper, uniform characteristics and performancein each laboratory’s respective platforms. In the AOL, it
was also necessary to convert the traffic scenarios into theformat appropriate for use in MACS. Python scripts werealso developed for use in the issuing of STA assignmentsin the runs involving scheduling constraints. Traffic
scenarios as well as convective weather representations

4.5 Airspace
The airspace used in this study consisted of test sectors
from the eastern portion of Kansas City Center (ZKC) andthe western portion of Indianapolis Center (ZID) in thecentral region of the United States. This was high altitude217

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall

were also developed in the AOL for use in the exploratory
runs.

traffic situations according to the allocation of functions inTable 1. This was done at two different traffic density
levels, which were approximately 1.5 and 2.0 times greaterthan current day (from here called Level A and B). The
two STA conditions involved the participants operating inthe same manner as in the Baseline conditions, but with theadded responsibility of maintaining the scheduled times ofarrivals for aircraft arriving at a number of area airports.
Automated support for this task was provided throughtimelines that were configured and assigned to each radar
controller workstation.

The data collection runs were conducted over the five days
following initial training. Each day was divided into threedifferent phases with the mornings devoted to thecollaborative short and medium runs. The first two runs of
the day involved the thirty minute scenarios with various
scheduling and traffic density conditions. These were thenfollowed by three, short 15-minute runs with varying STA
assignment conditions.
These morning runs were conducted in parallel in two
separate but adjoining rooms. Each room was equippedwith four radar controller stations-one for each test sectorand two area supervisor stations to be used by a single
supervisor. For consistency, each room was running thesame problems in the same conditions but was completely
isolated from one another. Within each room, the area
supervisor could monitor the traffic situation by walking
the floor and by assessing the flows and traffic load
statuses and predictions for each of the four test sectors via
the two supervisor stations. The four radar controllers,
however, were separated by partition (see Fig. 5) with thetwo ZKC test sectors on one side and the two ZID sectors
on the other. Inter-center communications andcoordination were required to be carried out via voicechannels and/or ground-to-ground data comm.

Data were collected from a variety of sources throughout
the study for later consolidation and analyses. During eachrun, screen recordings were taken on each of theworkstations. Every action performed within the simulation
network and the various states and aspects of the traffic
were recorded in real-time through the data collection
processes within MACS. Questionnaires were taken byparticipants after each run followed by a post-simulationquestionnaire once the final run was complete. A postsimulation debrief discussion was also conducted with thetest and confederate participants where a variety of topics
regarding the concept and the study as it was run werecovered.
5. RESULTS
While data were collected and a number of metrics wereanalyzed for each of the conditions in this study, thefollowing results section and ensuing discussion will focus
on the medium (M) duration runs. This portion of the study
was a 2 x 2 within subjects design where Traffic Density(NextGen Level A and B) was varied across Arrival Time
Constraints (Baseline and STA). The metrics to bepresented here will first be aircraft counts as they wererecorded for each sector, followed by reported workloadby the test participants, number of conflicts and conflict
resolution data, separation violations, and finallysubjective feedback from the participants.

During the runs, the tasks of the control team weredifferent in many respects from what they are today. As
this was a functional allocation study of ground-based
automated separation assurance, the main departures fromtoday’s roles and responsibilities were in this area. Table 1
provides a delineation of the tasks performed by theautomation and the human controllers during the datacollection runs.
For the medium duration runs, there were four conditions
in which participants operated. In the two Baseline
conditions, participants managed the automation and

Table 1 Allocation of functions between the automation and controller

Automation

Controller

Detect Separation Conflicts

Supervise the automation

Resolve trajectory-based conflicts (if within tolerances)

Resolve trajectory conflicts flagged by the automation

Resolve all time-critical traffic conflicts

Monitor and maintain schedule compliance

Alert controller to urgent problems

Place aircraft back on trajectory following automatedtactical maneuvers

Provide trajectory planning assistanceUse datalink to communicate

6218

ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)

5.1 Aircraft Count
Data for aircraft count are shown as a means of providing
the context for the remainder of the results. Fig. 7 shows
the average aircraft count in each of the four test sectors
plotted over time for the NextGen Level A and B traffic
densities.

Figure 8 Mean reported workload per condition (+/- 1 standard error)(SD = 0.74). For Traffic Density NextGen Level B, theBaseline-Level B condition had a mean reported workload
rating of 2.01 (SD = 0.79) and the STA-Level B had amore noticeable increase in mean reported workload ratingwith 2.34 (SD = 0.79).

Figure 7 Mean aircraft count per test sector over time

For aircraft count in NextGen Level A, the lower of thetwo densities, mean aircraft count in ZKC90 for the entire
run was 34.69 aircraft in sector (SD = 4.51), followed by
ZID81 with a mean number of 25.94 aircraft (SD = 5.29)
in sector. Sector ZKC 98 had a mean number of 20.86
aircraft (SD = 2.20), followed by ZID80, which had thelowest mean aircraft count with a mean number of 13.91
aircraft (SD = 3.71). Sectors ZKC90 and ZID81 were thelargest in terms of airspace volume of the four.

To investigate the differences in mean reported workload
ratings, a one-way, repeated measures Analysis of
Variance (ANOVA) test was conducted first to test thedifference between the Baseline and STA Arrival Time
Constraints conditions. While Fig. 8 and the descriptive
statistics showed the STA conditions being reported as
generally higher in workload than the Baseline conditions,
results from this test did not reveal any significant
difference (F(1, 31)= 2.60, p> .05). An additional test was
performed to examine the differences between the two
Traffic Density levels in mean workload ratings. This test
did yield significant differences where mean workload
reported in Traffic Density Level B was significantly
higher than in Level A (F(1, 31)= 14.85, p< .01).
Also of interest in the analysis of reported workload was
how it may have differed between sectors considering thatthe traffic complexities and characteristics differed
between them. Fig. 9 shows how the overall workload
differed between the sectors with ZKC 90 having thehighest mean workload at 2.56 (SD = 0.50) followed by
ZKC 98 at 2.09 (SD = 0.49), ZID 81 at 1.80 (SD = 0.33),
and ZID 80 showing the lowest mean workload at 1.49
(SD = 0.49). To investigate these differences further, a one
way ANOVA was conducted where a significant difference
in mean reported workload between the sectors was found(F(3, 60)= 7.65, p< .01). As a follow-up, Tukey’s Honestly
Significant Difference (HSD) tests were conducted where
it was found that ZKC90 had significantly higher workload
than ZID81 (p< .01) and ZID80 (p< .01). The only sector
that ZKC90 did not significantly differ from was ZKC98.

The NextGen Level B traffic density, found on the righthand side of Fig. 7, had higher mean levels of aircraft
count in each of the sectors relative to that experienced in
Level A. While the actual counts differed between theconditions, the order of sectors in terms of aircraft countheld constant across the Level B conditions with ZKC90
clearly having the highest mean aircraft counts with 45.76
aircraft over time (SD = 5.90), followed by ZID81 with amean of 34.66 aircraft (SD = 5.62), ZKC98 with a mean of
28.00 aircraft, (SD = 4.10), and ZID80 with a mean of
20.34 aircraft (SD = 4.75).
5.2 Workload
Throughout each run, workload prompts were displayed tothe participants at three minute intervals through aworkload keypad integrated into their display. The
prompts consisted of the keypad highlighting at theappropriate time accompanied by an audible tone. The
ratings were on an interval scale from one to six, with one
translating to the lowest level of workload and six thehighest. The analysis of workload was performed acrossTraffic Density Levels (A and B) and Arrival Time
Constraints (Baseline and STA). Fig. 8 presents the mean
workload ratings for each of these conditions.
The descriptive statistics show that for Baseline-Level A,
the mean reported workload was 1.78 (SD = 0.64) and thecorresponding STA-Level A was slightly higher at 1.83

219

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall

examined where it was also shown that Traffic Density
Level A resulted in significantly fewer numbers ofconflicts than in the higher density Level B (F(1,7)=492.80, p< .01).
5.3.2 Conflict resolutions
In addition to the number of conflicts, an understanding of
how the conflicts were resolved or attempted to beresolved was desired. This involved an examination of allof the clearances sent to aircraft in conflict. There were
three types of resolutions that could be sent via data comm:strategic automated resolution (Automated Resolution),
uplink by controller of resolutions deferred by theautomation (ATC Resolution), and the tactical TSAFE
maneuvers sent to aircraft with two minutes or less to lossof separation (TSAFE).

Figure 9 Mean overall workload per sector (+/- 1 standard error)

5.3 Conflict Data
To better understand the environment and operations as
they played out in this study, the number of conflicts was apoint of interest for analysis particularly as it lies at theheart of separation assurance.

Fig. 11 presents these data for each of the conditions where
naturally the overall number of resolution clearances sent
resembles the conflict numbers shown in Fig. 10.
From Fig. 11 one can see that, as designed, the majority ofconflicts were resolved by the strategic conflict resolution
automation. Upon further inspection of the strategic
resolutions, increases in the contributions of ATC
resolutions to the overall number were found as the traffic
density level increased. In Baseline-Level A, ATCresolutions accounted for 14% of strategic resolutions.
This increased to 21% of strategic resolutions in Level B.
This was likewise the case for the STA conditions where at
the Level A density ATC resolutions accounted for 20% of
strategic resolutions and increased to 25% in Level B.

5.3.1 Number of conflicts
Fig. 10 shows the mean number of conflicts recorded thatwere predicted to lose separation within any of the four test
sectors. From this figure one can see two visible trends:
The STA conditions resulted in fewer conflicts in the testarea regardless of traffic density, and overall the Level B
density had a greater number of conflicts than Level A.
Descriptive statistics show that the mean number ofconflicts for Baseline-Level A were 68.75 (SD = 4.19) andlower for STA-Level A with 58.00 (SD = 2.83). BaselineLevel B had a large increase with a mean number of
113.00 conflicts (SD = 3.74). STA-Level B showed afewer number of conflicts with a mean of 101.75 (SD =6.99).

A similar trend is also observable when accounting for
TSAFE resolutions and their contribution to the overall
number of clearances. For the Baseline conditions, there isan increase in TSAFE’s contribution from 12% to 22% astraffic density increases from Level A to B. This is also thecase for the STA condition where there is an increase from
16% to 24% from Level A to B.

Figure 10 Mean number of conflicts predicted to lose separation within
the test area (+/- 1 standard error)

A one-way, repeated measures ANOVA was first run to
examine the differences in the mean number of conflicts
between the Baseline and STA conditions where it was
found that the STA conditions did indeed result insignificantly fewer conflicts (F(1,7)= 30.52, p< .01). The
differences between traffic density levels were also

Figure 11 Mean numbers of Automated, ATC, and TSAFE resolutions
per condition

To better understand the composition of the clearance
numbers in terms of each sector’s contribution, the overall
number of clearances was separated out accordingly. These

8

220

ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)

mean numbers reflect an overall view of resolutions sent
throughout the study, regardless of condition.

place. From this figure one can see that the Level A traffic
density resulted in one OE each for the Baseline and STA
conditions. This number increased as traffic density
increased to Level B where both Baseline and STA
conditions resulted in a total of six OEs each with four andtwo additional PEs respectively.

Fig. 12 presents the mean number of all conflict
resolutions sent per sector. The most striking result fromthis portion of the analysis is sector ZKC 98’s share ofconflict resolutions relative to the other sectors. For
example, ZKC 98 had a mean number of 111.75 total
resolutions sent while ZID 81 sent a mean total of 86.00
resolutions. ZKC98 had required a greater number ofresolutions sent despite the fact that it had a lower level oftraffic (see Fig. 7). This is due in part to the smaller size of
ZKC 98 but, more importantly, also to the complexity ofthe sector. A more striking example of how the complexityof ZKC 98 affected the composition of clearances sent is acomparison of the mean number of TSAFE resolutions. All
things being equal, one might expect each of the four
sectors to be responsible for 25% of the total number of
TSAFE resolutions sent. However, as shown in Fig. 12,
this was not the case. Of the total mean number of TSAFE
resolutions sent, ZKC 98 accounted for 55% whereas theother sectors ranged between 12% and 17%. The
percentage for ZKC 98 was a direct result of its complexity.
The implications this may have had on workload andacceptability will follow in the Discussion section.

Figure 13 Total numbers of Operational Errors and Proximity Events percondition

As shown in Fig. 14, examining the number of separation
events by sector rather than condition revealed that ZKC98
had the highest number of OEs with seven followed by
ZKC90 with five and ZID80 with three; ZID81 had none.
In terms of PEs, however, ZID80 had the highest number
with three while the other sectors each had one.

Figure 12 Mean numbers of Automated, ATC, and TSAFE resolutions
per test sector

5.4 Losses of Separation
The benchmark for safety in any air traffic environment,
excluding mid-air collisions, is loss of separation events.
These events can be broken down according to twoclassifications: Proximity Events (PE) and Operational
Errors (OE). The PE classification, the less severe of thetwo, refers to events where a pair of aircraft has a closest
point of approach (CPA) that is between 8.33 km and 9.26
km (4.5 and 5.0 nautical miles (nm) respectively) laterally
and 243.84 m (800 ft) vertically. Operational Errors
involve aircraft pairs that have a CPA of less than 8.33 km(4.5 nm) laterally and 243.84 m (800 ft) vertically.

Figure 14 Total numbers of Operational Errors and Proximity Events per
test sector

5.5 Subjective Feedback
Following each run and at the conclusion of data collection
questionnaires were distributed to the participants in order
to elicit their subjective feedback on a range of topics.
References [12] and [14] provide additional data from thisfeedback. This section will focus on a subset of the
responses as they relate to situation awareness,
acceptability, and overall impressions of the concept.
5.5.1 Situation AwarenessThe concept as it was implemented and the environment inwhich it was introduced was a significant departure fromwhat the participants experience today. Because of that, theimpact of operations on their situational awareness was of

For this part of the study, there was a combined total of 20
PEs and OEs. Fig. 13 presents the breakdown of thesenumbers according to the conditions in which they took221

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall“Traffic level really not that much of a factor. Number ofconflictions at any one moment were more of a factor.”

interest. To gain insight into the impact, the Situation
Awareness Rating Scale (SART) [15] was used after eachrun where participants were asked to give responses tothree questions on their understanding of the situation,
demand that it placed on them, and their supply of
attention capacity during the run. Participants answered thethree SART questions on a 7-point scale from 1= “verylow” to 7 = “very high”. The responses to these questions
are then converted into an overall rating of situation
awareness.

5.5.3 Overall Concept Impressions
Following the completion of the data collection phase of
the study, a post-simulation debrief and discussion was
conducted. This allowed both the researchers andparticipants to converse openly and freely regarding theirideas and opinions on the concepts presented and how they
were carried out in the study. With respect to the concept,
participants provided positive feedback with quotes suchas, “you’re on the right track,” and, “it seems fairly natural,
why not do it?” However, some concern was expressed
regarding some of the more technical aspects of how theconcept could be realized. One participant remarked, “…itseemed as if controller and automation fought against eachother at times to resolve conflicts.” This was mainly inresponse to the hands-off approach taken in the operationof automated tactical conflict resolutions. Despite thesetypes of concerns, however, the overall mood andassessment of the concept was positive and can best be
summed up by one of the participants who said, “It’sinevitable, I think the concept is strong, it needs work andtesting, I think it’s the way we’re going to go.”All three questions were answered using the full range ofthe scale – from 1 to 7- with a good spread of answers
particularly between the sectors. As shown in Fig. 15,
answers for the understanding question tended to be at thehigher end of the scale with few lower (“poorunderstanding”) responses. Overall, participants felt they
had a “very good understanding” of the situation (M=6.33,
SD=1.37), an “average” supply of attention capacity(M=4.25, SD = 1.89), and that there was only “a little”
demand on their attention (M=2.96, SD = 1.37) (see Fig.
15). These responses combine to indicate that participants
felt they had a “reasonable situation awareness” (M=5.08,
SD=2.28).

6. DISCUSSION AND CONCLUSIONS
The results presented in this paper were from a larger study
that examined the functional allocation of ground-based
automated separation assurance, which was also part of an
overall study that examined the functional allocation of
separation assurance between the flight deck and a groundbased ATC system. These studies were part of a continuing
series of investigations into the realm of automatedseparation assurance and its related issues particularly as
they relate to human-systems integration.
The results presented here were based on the medium
duration runs of the larger study that were 30 minutes inlength and varied Arrival Time Constraints (Baseline andSTA) across two levels of Traffic Density (NextGen Level
A and B). With the exception of number of conflicts where
the STA condition had significantly fewer conflicts thanBaseline, comparisons between Baseline and STA did notyield many insightful results. The most noteworthy
comparisons were between the Traffic Density levels.

Figure 15 Mean SART ratings compared by the level of traffic density

5.5.2 Acceptability of Operations
Subjective feedback was also sought regarding theacceptability of operations as they were experienced in thisconcept’s environment. This was gained through thepresentation of questions closely aligned with theController Acceptance Rating Scale (CARS) [16].
Participant answers are compiled on a scale from one toten where one indicates that the operation is not safe andten indicates the system is acceptable. Participants rated
the separation assurance operation as safe less often under
NextGen Traffic Level B (67.5% of the time) than under
Level A (90.6% of the time). A general analysis of thecomments indicated that it was not the absolute volume oftraffic that concerned participants but that the situations
became more complex as the traffic increased (and finding
a conflict solution with a clear path became less and lesseasy). For example, one participant commented that,In Fig. 7 the plots of sector counts over time for each of
the test sectors showed that NextGen Level B had
generally higher sector counts but that the differences
ranged between the sectors. These plots provided an
important context in which to view the subsequent results
because what was witnessed was a shift away from sector
count alone being the driving factor for workload andsafety to other factors such as local complexities within asector.
For example, traditionally one might expect workload
ratings to be highest for the sectors with the highest sector
10222

ENRI Int. Workshop on ATM/CNS. Tokyo, Japan. (EIWAC 2010)

counts, which were, in decreasing order, ZKC90 and
ZID81, the two largest sectors. However, an examinationof reported workload revealed that it was ZKC98 that
resulted in higher workload than ZID81, despite lower
levels of traffic. It should be mentioned that although there
were inter-sector differences, even the highest overallmean workload ratings were still relatively low and within
an acceptable range.

Subjective feedback from the participants provided thisresearch effort with a different perspective on operations
apart from the objective data. Situation awareness was
rated as being “reasonable” even with the levels of
automation in operation but that it was not the same for all
sectors. In terms of acceptability, participants rated
operations at the lower traffic density quite highly.
However, this showed a reversal as the density increasedsuch that there was a 23% decrease in participant ratings.
This did not seem to affect their overall assessment of theconcept, however, where the general consensus was thatthe approach taken was a viable way forward.

In this simulation, ZKC98 was at once one of the smallest
sectors while also being the most complex. This
complexity existed by virtue of heavy arrival and departure
corridors within the sector where transitioning aircraftinteracted with aircraft at level flight. It was this type ofinteraction and inherent complexity that allowed for theobservation of the equal or greater impact of complexity
over that of simple sector count on operations andacceptability.

To conclude, in this study it was found that the increase intraffic density resulted in the greatest impact on operations.
However, it was not aircraft count alone but local
complexities that affected the observed results. Thisobservation coupled with feedback provided by theparticipants speaks to the promise that this functional
allocation concept of ground-based separation assurance
holds for the ability of the NAS to accommodate theenvironment envisioned as part of NextGen.

This was first evident through the examination of conflictdata where it was initially found that as traffic density
increased from Level A to B, there was a significant
increase in the number of conflicts. This naturally resulted
in a greater number of conflict resolutions to be issued.
Upon further examination of clearances by sector it was
found that despite ZKC98 having the third lowest level oftraffic count, it had the highest number of resolutions sent.
This was particularly the case in terms of tacticalresolutions where more than half of all sent were in this
sector.

7. ACKNOWLEDGMENTS
We greatly appreciate the support and collaboration of theNASA Airspace Program, the FAA, and the separation
assurance research team at NASA Langley. Thanks also go
to the MACS development team at NASA Ames.
8. REFERENCES[1] Concept of Operations for the Next Generation Air

For sectors ZKC98 and ZKC90, the number of operational
errors followed the same trend as the number of clearances
sent respectively. This was not the case, however, with
ZID81, which had the second highest levels of traffic andthe third highest number of clearances sent but did nothave any resulting operational errors.

Transportation System Version 3.0.
[Online] October 1, 2009.
http:/www.jpdo.gov/library/NextGen_ConOps_v3.pdf.
[2] T. Prevot, T. Callantine, J. Mercer, and J. Homola,“Human-In-the-Loop Evaluation of NextGen
Concepts in the Airspace Operations Laboratory”.
AIAA MST Conference, Toronto, Canada.

ZID81 had the second highest levels of traffic but was
managed such that operations and workload were tenablewith the least impact on safety with only one proximity
event. This speaks to the potential that greater levels of
traffic can be safely accommodated but also that particularattention needs to be paid to the complexity of the local
environment and the ability to mitigate or manage it
effectively.

[3] FAA Aerospace Forecasts FY 2009-2025

http://www.faa.gov/data_research/aviation/aerospace_
forecasts/20092025/media/FAA Aerospace Forecasts
FY 2009-2025.pdf[4] H. Erzberger, “The Automated Airspace Concept”.

4th USA/Europe Air Traffic Management R&D
Seminar, Santa Fe, NM, USA, 2001.

While the ultimate goal of any system is to eliminate
separation violation occurrences, they did occur here andare being investigated. Preliminary results show that latedetection and the difficulties related to accurate trajectory
predictions for transitioning aircraft played significant
roles in the violation of separation minima. Despite thecomplexities described and the number of operational
errors, while attempting to assess the success rate of theconflict resolutions sent it was found that over 99% ofconflicts were successfully resolved and avoided an
operational error in this portion of the study.

[5] H. Erzberger, “Transforming the NAS: The Next

Generation Air Traffic Control System,” 24th ICAS,
Yokohama, Japan, August 2005.
[6] J. Homola, “Analysis of Human and AutomatedConflict Resolution Capabilities at Varying Levels ofTraffic Density”. (Master’s Thesis). San Jose State
University, San Jose, California, 2008.
[7] T. Prevot, J. Homola, and J. Mercer, “Human-in-the-Loop Evaluation of Ground-Based Automated223

J. Homola, T. Prevot, J. Mercer, C. Brasil, L. Martin, C. Cabrall

Separation Assurance for NextGen”. ICAS 200811.4.5, and AIAA-ATIO-2008-8885, Anchorage,
Alaska, 2008.

[14] T. Prevot, J. Mercer, L. Martin, J. Homola, C. Cabrall,and C. Brasil, “Functional Allocation for GroundBased Automated Separation Assurance in NextGen”.
HCI-Aero 2010, Cape Canaveral, Florida, 2010 (inpress).

[8] T. Prevot, J. Homola, J. Mercer, M. Mainini, and C.

Cabrall, “Initial Evaluation of NextGen Air/Ground
Operations with Ground-Based Automated Separation
Assurance”. ATM2009, Napa, California, 2009.

[15] R. M. Taylor, “Situational awareness rating technique(SART): The development of a tool for aircrew
systems design”. In “Situational Awareness in
Aerospace Operations” (AGARD-CP-478; pp 3/13/17), Neuilly Sur Seine, France, NATO-AGARD,
1990.

[9] H. Erzberger and K. Heere, “Algorithm andoperational concept for solving short range conflicts”.
ICAS 2008-8.7.5, Anchorage, Alaska, Sept 15-19.
[10] J. Homola, T. Prevot, J. Mercer, M. Mainini, and C.

[16] K. K. Lee, K. Kerns, R. Bone, and M. Nickelson,

Cabrall, “Human/Automation Response Strategies in
Tactical Conflict Situations”. DASC 2009, Orlando,
Florida, 2009.

"Development and Validation of the Controller
Acceptance Rating Scale (CARS): Results of
Empirical Research," 4th USA/Europe Air Traffic
Management Research and Development Seminar(ATM-2001), Santa Fe, NM, 3-7 December, 2001.

[11] R.A. Pearce, “The NextGen JPDO Model ofInteragency Planning”. AIAA 2009-7010, 9th AIAAATIO, Hilton Head, SC, 21 - 23 September 2009.
[12] D. Wing, T. Prevot, J. Murdoch, et al. “Comparison of

COPYRIGHT

Airborne and Ground-Based Functional Allocation
Concepts for NextGen Using Human-In-The-Loop
Simulations”. AIAA, 10th ATIO, 2010 (in press).

The authors confirm that they, and/or their company or
institution, hold copyright of all original material includedin their paper. They also confirm they have obtained
permission, from the copyright holder of any third party
material included in their paper, to publish it as part oftheir paper. The authors grant full permission for the
publication and distribution of their paper as part of the
EIWAC 2010 proceedings or as individual off-prints fromthe proceedings.

[13] T. Prevot, “Exploring the Many Perspectives ofDistributed Air Traffic Management: The MultiAircraft Control System MACS”. In S. Chatty, J.
Hansmann, & G. Boy. (Eds). HCI-Aero 2002, AIAA
Press, Menlo Park, CA. pp 149-154.

12

224