U.S. Government work not protected by U.S. copyright

Human-system interaction issues and proposed 

solutions to promote successful maturation of the 

UTM system 

Cynthia Wolter 

SJSURF 

Lynne Martin 

NASA Ames 

Kimberly Jobe 

SJSURF 

at NASA Ames Moffett Field, USA at NASA Ames 

Moffett Field, USA lynne.martin@nasa.gov Moffett Field, USA 

cynthia.wolter@nasa.gov 

Abstract— Over five years, NASA, together with partnering 

organizations, has been developing and successfully 

demonstrating the maturing capabilities of the Unmanned 

Aircraft Systems (UAS) Traffic Management (UTM) system and 

its ability to support communication and coordination among 

small UAS operations through a series of flight tests. During these 

flight tests, human-system interaction (HSI) elements were also 

explored in order to identify the barriers to implementation as 

human operators transitionally fulfill roles that will be ultimately 

tasked to future automation. Throughout the tests, similar issues 

were regularly documented and are expected to persist if not 

formally addressed by consistent procedures, intuitive design, or 

regulation. Documented here, along with suggested mitigations, 

are the most frequently noted HSI items, which include operator 

training, data standardization, and information quality. 

Keywords—small Unmanned Aircraft Systems (sUAS), UAS 

Traffic Management (UTM), future airspace management 

I. INTRODUCTION 

For the last five years (2015-2019), the Unmanned Aircraft 

Systems (UAS) Traffic Management (UTM) research project

has been developing and testing concept ideas for enabling small

UAS (sUAS) operations in low altitude airspace (i.e., ground to 

400 feet) [1]. To do this, a series of incrementally complex flight 

test demonstrations, from rural farm-field flying to enacting 

contingency maneuvers in a downtown city area, were run at a 

total of eleven different test locations. The demonstrations 

resulted in over a thousand data collection flights using 89 live 

UAS (both multi-copter and fixed wing) with nearly seventy 

flight crews participating. During the later demonstrations, 

simulated vehicles were included in scenarios to test the system. 

By the end of the series, more than ten Unmanned Aircraft 

Systems (UAS) Service Suppliers (USSs) had taken part to 

manage flights’ operational volumes and provide the ground 

control station (GCS) anchors for the UTM system. Each flight 

test successfully demonstrated a different layer of capabilities 

of 

kimberly.k.jobe@nasa.gov 

the system with scenarios that had been scoped to set up sets of 

sUAS corner-case events to test new functionality. 

The UTM system was developed in response to the need for

an airspace system to manage the projected increasing usage of 

sUAS for aerial activities, especially in commercial applications

[2]. It is a federated system that allows many operators to bring 

and link in their own USS which represents their own operations 

and facilitates participation in UTM. The system needs to 

facilitate automated communication and coordination between 

sUAS operations as the existing air traffic control systems, 

responsible for 5,000 daily airplane flights, could not feasibly

monitor the millions of sUAS forecast to come into use in a nearterm time-frame [3]. 

An aim of the UTM system is that it will ultimately operate 

in a largely automatic manner, relying less frequently on 

interaction from a human operator. Until that substantiallyautomated end state has been safely reached, thought needs to 

be given to how humans can best perform those tasks that need 

to be manually completed and interact with UTM (i.e., humansystem interaction or HSI) in the interim. During this time, the 

efficiency and effectiveness of the system will depend on the 

level of workload, attention, and awareness required of the 

human operator. 

The UTM research project advanced through a risk-based 

approach, where both the risks of the environment and the 

complexity of UAS operations encompassed four different 

technical capability levels (TCLs) (Fig. 1) [4]. The research and 

development was designed to advance through a series of five 

flight tests, from least through more complex and riskier 

environments, to explore the performance of the system. The 

first capability level, or TCL1, involved an environment that was 

assessed as low-risk due to the flying domain being 

characterized by open spaces with few structures and people. 

Operations during this time were straightforward and 

notification-based. As environmental complexity and risk 

increased in TCL2 through to TCL4, sUAS were operating over 

urban areas with closely-spaced structures of varying height, 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

NASA UTM planned technical capability level progression. 

dense population, and many dynamic obstacles below (e.g., cars 

and people). For these operations, sUAS operators have to take 

airspace complexity into account in their flight planning and 

operations and prepare a series of contingency plans for a variety 

of different potential events, like loss of power or loss of 

communication with the vehicle. 

During the five flight demonstrations of the prototype 

system, which were defined according to the level of the risk 

model that they were demonstrating, functions were frequently 

allocated to the human operator, while more functions 

transitioned from manual to automated as the system was 

developed. As the UTM system matures further, it is expected 

that the role of the human operator will change to being more 

supervisory, and therefore the nature of the HSI will also change. 

For all of the flight tests, attention was paid to user 

information requirements, and the primary users were the 

ground control station crews including the UAS pilot-incommand (PIC), GCS operator, and UAS engineers. The subject 

of this paper is a metadata analysis that aims to examine the 

consistent themes that arose from the qualitative data collected 

from these teams across these five years of events. 

II. METHODS 

During the four most recent sets of flight tests, data was 

specifically collected from the operators, including all members 

of the flight crews, about their experiences using UTM and their 

USS clients. Flight crews, the PIC in particular, often held 

manned aviation certifications, bringing their general or military 

aviation experience to meet the challenges at each test site. In 

addition, crews increasingly had UAS mission experience and 

many UAS flight hours. Qualitative data were collected through 

surveys, observations, and group interviews which were then 

compiled to describe the experience of operating within UTM. 

The first flight test demonstration (Test 1), that was 

exploring TCL1 conditions (Fig. 1), was held over eight days, 

spread across a calendar year from August 2015 to September 

2016. It tested three fundamental sUAS traffic management 

services: user authentication, flight planning, and information 

services in an open space area where environmental complexity 

was low (i.e., no structures and no population). Vehicles flew a 

series of box patterns with the UTM services in operation. In 

total, ten different aircraft flew more than 100 flights during the

eight days. As the focus was the testing of UTM services, crews 

were not asked for their feedback as users in a structured way, 

only through informal discussions. While the flight test 

demonstrated the feasibility of the three focus services, it also 

drew attention to other important areas, such as the need for 

common situation awareness (SA) among all stakeholders. As a 

result, the following flight tests consistently included inquiry 

into the user experience and information requirements from a 

user-perspective, and it is only these data that are the focus of 

the summary analysis in this paper. 

A. Summary of data sources 

The second demonstration (Test 2) increased the level of 

complexity of the flight tests by inviting both new and existing 

partner organizations to operate together at a test site in Reno, 

Nevada (Fig. 2) under TCL2 conditions. The aim was to fly 

scenarios where there were multiple simultaneous sUAS in 

operation using the UTM system. The flight test took place 

during nine consecutive days in October 2016. Four scenarios 

were developed to script multi-vehicle, altitude stratified 

BVLOS (beyond visual line of sight) operations1

. Across the 

range, five GCS locations were set up at different points around 

the test site. Eleven flight crews each brought their own vehicle 

to the flight demonstration. These eleven crews were divided 

into three separate groups of four or five crews (three crews 

participated twice to enable this). These three groups attended 

the flight demonstration for three days each: The first day 

consisted of an initial briefing and time for the crews to set up 

and test their equipment, while the second two days had the 

crews flying proficiency flights and a selection of test scenarios. 

1 Due to test and vehicle constraints most flights were visual line of sight 

operations. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Generally, crews flew two proficiency flights and two scenarios 

per flying day, generating 36 sUAS data collection flights, 

although the third group had one flight day cancelled due to high 

winds. Over the five data collection flying days, 72 participant 

survey responses were collected, five end of day group debriefs 

were held, and researcher observations were conducted during 

the operations of all eleven crews. 

Location of the second UTM flight test demonstration (Test 2) under 

TCL2 with ground control stations GCSs marked. 

In 2017, the third demonstration (Test 3) invited six testpartners to host flight tests at their local testing sites, involving 

other partners of their own choosing, and they were given the 

ability to tailor the test scenarios to the features and 

environments of their respective sites. All sites had few 

structures, were sparsely populated and were considered TCL2. 

Each site organized a shakedown followed by approximately a 

week of flying for data collection – yielding a total of 17 flying 

days2 over eleven calendar days. The demonstration resulted in 

over 270 data collection flights flown by 18 flight crews using 

27 live rotorcraft, and seven USSs. Test sites were asked to 

incorporate six advanced flying challenges, such as BVLOS 

and stratified operations, with a selection of five complexityincreasing events, such as contingency maneuvers. These 

challenges were incorporated by partner sites into scenarios that 

they created, which took advantage of the unique features of the 

terrain and conditions at their of their respective test sites. Sites 

created a total of 17 test scenarios. Over the 17 days when 

flights were conducted, 141 participant survey responses were 

collected, 18 end-of-day group debriefs were held and the 

research team made 34 sets of observations of flight crews. 

During 2018, the fourth demonstration (Test 4) was 

conducted by the same six organizations as Test 3, flying out of 

four new locations and three of the same locations as the 2017 

tests. They completed 33 days of flying in a March to May flight

window across the six sites. The demonstration generated over 

800 data collection flights using 28 live rotorcraft, piloted by 17

flight crews, linked into the UTM system by nine USSs. All sites 

flew a subset of 20 scenario outlines that had been scoped by the 

UTM project. These short vignettes, or performance tests, 

addressed one of the four themes of key concern for the third 

level of UTM operations: exploring Communication, 

Navigation and Surveillance (CNS) issues, Sense and Avoid 

(SAA) tools, data and information exchange routines, and tests 

that exercised procedural aspects of the UTM concept. For 

example, the “CNS1” test involved using redundant command 

and control (C2) and explored whether control could be 

maintained of the sUAS when one communications link was 

lost. By contrast, tests like “Concept 2,” required sites to create

an event that included contingency maneuvers [5]. Over the 50 

flight-days, 274 participant survey responses were collected, 22 

end of day group debriefs were held, and the research team 

observed multiple crews during over 50 flight sessions. 

In 2019, the fifth demonstration (Test 5) comprised two 

flight tests that lasted ten days each; these occurred in the states 

of Texas and Nevada. These flight tests were focused on the 

most complex environments thus far in which sUAS are likely 

to fly (i.e., densely populated urban settings) as shown in Fig. 3, 

and events that may be critical in urban city locations, such as 

avoiding dynamic obstacles, landing in close proximity to 

buildings, and dealing with system failures. This demonstration 

resulted in over 700 data collection flights using 14 live 

rotorcraft and 30 simulated vehicles which were operated by 17 

flight crews and seven USSs. The approach was designed for a 

Ç 

Ç 

Ç 

Ç 

Ç 

Fig. 3. One location for the fifth UTM flight test demonstration (Test 5, 2019) 

under TCL4 with GCS marked. 

2 Each day of flying for each site was counted as a separate “flying day” even 

though one site may have flown on the same calendar day as another site. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

TCL4 level of risk, demonstrating events that may be critical in 

city locations, such as avoiding dynamic obstacles, landing in 

close proximity to buildings, and dealing with system failures. 

This demonstration resulted in over 700 data collection flights 

using 14 live rotorcraft and 30 simulated vehicles flown by 17 

flight crews and seven USSs. 

The approach was designed with five scenarios that set up 

diverse sets of UAS events and activities. These scenarios 

focused on a variety of potential events and issues, some of 

which were an incoming weather front, sharing airspace, a USS 

failure, and multiple vehicles experiencing CNS issues. The test 

site was required to complete three executions of each scenario. 

Over the 19 days when flights were conducted, 149 participant 

survey responses were collected, 19 end of day group debriefs 

were held and the research team observed over 75 flight 

sessions. 

This document presents a summary of the qualitative data 

collected over the four demonstrations from October 2016 to 

August 2019, i.e., Test 2 to Test 5. The common types of data 

were participant surveys, group debriefs, and researcher 

observations (Table 1). These were developed by considering 

common human-system interaction themes, including user 

information requirements, user decision making, user situation 

awareness, and communication. Although the methods of data 

collection and themes were the same, the focus questions for 

each demonstration differed. The survey questions, discussions 

in group debriefs, and researcher observations were tailored to 

the scenario events that were being exercised and hence are not 

specifically comparable. The summary below explores common 

threads between the main findings across the most recent four 

years of data collection (excluding the first year, which explored 

the Test 1 study with TCL1 characteristics) . 

TABLE I. QUALITATIVE DATA COLLECTION SUMMARY ACROSS FIVE UTM FLIGHT DEMONSTRATIONS. 

Data collection test name 

Flight demonstration level and year 

Test 1 Test 2 Test 3 Test 4 Test 5 

Technical Capability Level TCL1 TCL2 TCL2 TCL3 TCL4 

Date 2015-16 2016 2017 2018 2019 

Data collected 

Participant surveys N/A 

72 asked 

at the end 

of flights 

141 online 274 online 149 online 

Interviews/ debriefs N/A 5 18 22 19 

Hours of debrief N/A Approx. 

5 hours 

Approx. 

9 hours 

Approx. 

8.5 hours 

Approx. 

9 hours 

Flight sessions observeda N/A 24 34 50 75 

a. Flight sessions were observed by a team specifically focused on user interaction and experience of the operations. 

III. DISCUSSION 

The flight test demonstrations were intentionally designed to 

test the UTM system – specifically the architecture and the 

software components within it. However, assisted by real-world 

settings, the demonstrations also highlighted issues in the wider 

UTM environment that may need to be addressed. 

The flight tests successfully demonstrated that the UTM 

system is rapidly developing into a viable mechanism for 

allowing communication and coordination between increasingly 

numerous sUAS operations. The UTM system correctly 

collected an increasing amount of incoming information about 

operations, and USSs were able to retrieve the necessary 

information in order to inform their users about negotiations, 

their own and others’ operational states, potential conflicts, 

airspace restrictions, and other actionable items. However, the 

success of the system architecture demonstrated the need for 

further inquiry into ensuring that the human operator, interacting

with UTM through a USS, is able to effectively utilize all of the 

maturing capabilities of the system. The development of USSs 

progressed through the years, and the USSs that each operator 

used to interface with UTM varied within each series of flight 

tests. This allowed trends to emerge within the observations, 

particularly in regards to the amount of operator familiarity with 

UTM, the amount of information that was displayed, and the 

standardizations and procedures. 

A. System Level Issues – Operator Training 

For the first four flight tests (Test 1 to Test 4, see Table 1), 

the PIC role was filled by individuals who held a pilot rating 

(e.g., manned flight certificate such as for an instructor or a 

private pilot license [PPL]) in addition to the remote pilot 

certificate with sUAS rating (i.e., Part 107) [6]. These pilots

drew on their manned flying experience to communicate on the 

test range, scan for traffic, and plan flights with respect to 

weather. In Test 1 through Test 4, crews were able to understand 

the UTM procedures and usability complications with their 

USS, such as too little information, by drawing on their aviation 

background and because fewer factors needed to be considered 

for solution options due to the less complex environments. 

Funded through the UTM Project under the NASA Airspace 

Operations & Safety Program (AOSP). 
However, for the fifth demonstration (Test 5), the 

consequences of actions became more severe, as the 

environment was more complex. For example, there were many

more structures to be aware of, which resulted in fewer breakout 

options and generally an increased time pressure. One of the test 

sites in Test 5 took a different approach to their crews and 

trained a group of non-pilots to operate a fleet of sUAS. This 

approach provided a good opportunity to observe whether 

novice users of the UTM system had different needs or methods

for using UTM information. Novice 

9 

8 

Very 7 high 

6 

5

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean SA rating 

users required more 

explanation or detail around information presented by the UTM 

system. Notifications that were only two or three words or used 

UTM-system jargon were not always understood and there was 

insufficient time to look up information during operations. In 

4 

3 

addition, novice users did not always know the action-options 

that were intended in response to events, could not revert to Very 

2 

1 common aviation procedures (unlike those with a PPL), and did 

not understand the safety implications of some of the actionoptions in response to scenario events. This would suggest that 

additional elements of pilot training should include (accepted) 

procedures for reacting to a range of possible events. 

B. Information Quantity 

1) Too much information 

Throughout the flight tests, researchers noted instances of an

over-abundance of information. This was less apparent when the 

flight tests had fewer aircraft that were flying more nominal 

scenarios in rural environments, but became more so when the 

airspace was more densely occupied by other UASs and 

obstacles. Display clutter was first noted during the third flight 

test (Test 3) and became more common from there as the amount

of incoming information increased. Users could receive 

messages about the state of their own operation, other 

operations, conflicts, negotiations, airspace restrictions and 

emergency alerts, and were oftentimes over-saturated with 

information. For example, one participant in Test 5 said: “There 

were over 80 negotiations. I didn’t get a lot of info on each but 

if I did it would have been too much anyway. "Consequences of 

insufficient information filtering were that similar messages 

were sometimes ignored by crews if they seemed to be 

duplicates, non-persistent messages were sometimes missed if 

they were obscured by more recent messages, and the mental 

effort to maintain information currency was sometimes difficult. 

2) Too little information 

Researchers also observed users lacking information that 

they wanted or needed. In early tests when the USS client 

interfaces were less developed, some details were absent that 

users required in order to react quickly to messages, but this was 

a less critical issue as the operating environment allowed them 

time to obtain those details. As the tests advanced into more 

dense and urban environments, increasing the amount of 

information for each user to process, the USS interfaces also 

greatly matured to be more user-friendly, which is reflected in a

general increase in situation awareness over time. This is 

illustrated in Fig. 4, which shows the mean of participants’ 

responses to a question about their perceived SA across four 

flight tests, from 2016 (Test 2) to 2019 (Test 5). The general 

trend in terms of participants consistently rating their SA as 

low Test 2 Test 3 Test 4 Test 5 

(2016) (2017) (2018) (2019) 

Flight test demonstration 

Fig. 4. Mean situation awareness responses from flight demonstrations held in 

2016 and 2018. Response scale was 1-7, extended in chart to show deviations, 

n-Test 2 = 30, n-Test 3 = 27, n-Test 4 = 41, n-Test 5 = 14. Note: Survey 

questions were not identical across all flight tests. Only data from comparable 

questions have been included here. 

slightly higher every year to the later years, after the UTM 

system had been developed and improved for an additional 

period. However, users in these later tests had less time to act, 

and so the need for each message to contain more necessary 

information grew. Acting quickly on sub-optimal information 

seemed to lower users’ confidence in their decisions (Fig. 5) as 

they were often observed to seek out second opinions. Figure 5 

shows the mean of participants’ responses to a question about 

their confidence in the accuracy of information they received 

from UTM across two different flight tests, one in 2017 (Test 3) 

and one in 2019 (Test 5). During Test 5, participants gave, on 

average, lower (i.e., less confident) responses to these questions. 

Users’ awareness for how their own operation interplays with 

the surrounding environment could be improved by streamlining 

the incoming messages and alerts and by tailoring the included 

text to facilitate easy and quick comprehension. 

C. Standardization 

1) Uncommon terminology 

The language and commonly used phrases for messaging 

were observed as a source of confusion that manifested more as 

the user groups became diversified. Throughout the tests, users 

who had a hand in developing their USS clients did not struggle

with interpreting the information it displayed, and Test 2 to Test 

4 showed that these were the most common types of users. 

Later, as users with different backgrounds emerged, flight crews 

were sometimes confused about the information they were 

receiving. Equipping operators with shared terminology to use 

while flying in the UTM airspace could increase the efficiency 

with which they conduct those operations. 
8

Very 7 high 

6

5

4

3

2

Very 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

low Test 3 (2017) Test 5 (2019) 

Flight test demonstration 

Fig. 5. Mean confidence in accuracy of information responses from flight 

demonstrations held in 2017 and 2019. Response scale was 1-7, extended in 

chart to show deviations, n-Test 3 = 42, n-Test 5 = 65. Note: Survey questions 

were not identical across all flight tests. Only data from comparable questions 

have been included here. 

2) Measurement consistency 

Across trials, systems or operators varied in terms of 

methods used to calculate altitudes or used different units of 

measurement entirely (see [7]). Some systems were using AGL 

(height above ground level), while others were using MSL 

(mean sea level) or WGS84 (World Geodetic System 84) as their 

standard. This was especially problematic and slowed the pace 

of operations when two or more sUAS were attempting to fly 

stratified, with one above another. When an operation was 

unexpectedly rejected by the UTM system due to an altitudeconflict, time had to be taken to resolve the issue. This occurred 

usually via voice communication, which is not intended as a 

common interaction in the UTM concept. An accepted standard 

for common measurements must be established to further 

increase the efficiency of UAS operations in the UTM working 

airspace. 

3) Undefined procedures 

Similar to the issue with uncommon terminology, users were

not generally observed to have a full understanding of how to 

respond to specific UTM events. The live flight demonstrations 

were designed so that the earliest tests looked at the robustness 

of the UTM system under nominal conditions (i.e., one or two 

vehicles flying within line of sight of their operators), and later 

tests introduced more off-nominal conditions (i.e., large-scale 

contingencies and emergency management). While these tests 

successfully demonstrated the UTM concept and the capabilities 

of the system under these stressors, it also exposed gaps in 

procedures for the human operator when interacting with UTM

through their USS client during these events. The absence of 

standard or established procedures was less concerning during

the first three tests considered in this paper due to the nature of 

the scenarios where operators had fewer planned events to react 

to and more time to strategize their actions. Mean confidence rating 

The later demonstration scenarios were designed to include 

many overlapping events occurring in densely populated urban 

areas, adding both urgency and complexity to the decision 

making process. As an example, there were multiple ways to 

respond to a vehicle emergency including: land immediately, 

directly and immediately return to launch, or retracing the 

original route to the launch point, either with or without creating 

a new UTM volume to remain within. In simple scenarios, 

operators had more time to make these decisions, and there were 

fewer nearby UASs or static obstacles to consider. Later, the 

need to proceduralize responses became more apparent as 

operators were faced with complex decisions. For example, one 

participant in Test 5 said: “We have arrangements ... to always 

decline re-plan negotiation requests and always accept 

intersection requests.” Not only would procedures help users 

make quick decisions and be confident about their own actions, 

they would also assist a user’s ability to predict the actions of 

other operations. 

D. UTM Beyond TCL4 

Given the successful progression, development, and 

demonstration of the UTM system, new generations of UAS 

operators, who are using UTM and flying their sUAS in public 

areas to serve local populations, will need to be able to interact

efficiently and effectively with the system to fully utilize its 

capabilities. These operators will need to be aware not only of 

the functioning of their vehicles but also how to manage their 

flight volumes through UTM and how to monitor for and 

interpret the activity of surrounding sUAS operations. Operators 

will need to be able to understand and respond to UTM alerts 

and warnings. 

The UTM system has a planned end state that is more 

automated than the prototype system tested during Tests 1 to 5. 

As tools are developed and the system is utilized, many 

functions will gradually transition from manual to automated. 

Until functions are safely automated, human operators will 

continue to complete these tasks. This leads to two broad 

challenges for human-system integration within UTM. 

Consideration needs to be given to the interaction required 

between the operator and UTM (HSI) to enable the operator to 

complete all the tasks they are required to fulfil as the system

moves through every stage of its development. As stated earlier, 

in the early fielding of the system, the level of operator 

workload, and attention needed to successfully operate while 

flying an sUAS will determine its efficiency and effectiveness. 

Tasks will naturally be coherent because operators are involved 

in every aspect of the system. As the UTM system matures, the 

nature of the HSI is predicted to transition to a state where the 

system completes many more of the operational functions and 

the operator moves into a more supervisory role [9]. As this 

process occurs, a second broad challenge arises, which is to 

consider what users need to know about the integration between 

automated tools so that they can find the information they need 

should they need to become a manual part of (intervene in) the 

system (e.g., for corner-case events where there is not yet an 

automated solution) [10]. 

To support different stages of HSI, information presented in

interfaces will need to be tailored. During initial operation of 

UTM, information about events and alerts needs to be clear and 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

timely as operators have to comprehend information in real-time 

to make decisions and take actions. As levels of automation 

increase and operators step into a more supervisory role, 

information presented to operators will need to provide a higherlevel view of operations – but one where the user can drill down 

into specific topics if the need arises. For all stages of 

automation, displays should be well-organized. 

IV. SUMMARY 

The series of UTM flight tests hosted by NASA across five 

years and 17 locations were successful demonstrations of the 

UTM system. The tests resulted in over 2000 data collection 

sUAS flights and involved 72 flight crews. The viability of the 

concept of a UTM network for sUAS traffic was successfully 

demonstrated. 

A thrust in the flight test research was to explore methods for 

the exchange of information. Although UTM itself was shown 

to support the communication and coordination between sUAS 

operations, through the years of data collection many HSI 

challenges and obstacles to successful operation were observed. 

Three of these categories were highlighted above: 

standardization, information quality, and considering the 

complete system during transition to automation. The specific 

challenges within these categories were present from the 

beginning of the research, and most became more apparent as 

both the operations and the operating environment increased in 

complexity. An operator’s ability to maintain a successful, 

efficient, and safe flight was sometimes challenged by a lack of 

universal standardization of measurement units and 

calculations, non-specific procedures, and by nonstandard(ized) terminology. A set of standard metrics will help 

operators ensure that they are maintaining safe distances and 

buffers. An over-abundance of information that was not always

pertinent to current tasks also posed challenges, as it could 

clutter the displays and potentially overwhelm the user, leaving

them unable to locate the information they desired or to 

potentially miss important notices that were relevant to their 

own operation. Conversely, having too little information was 

also a cause for concern, like when pertinent messages did not 

contain enough context to be immediately usable, and this 

necessitated that the operator spend time looking for the missing 

information elsewhere. These issues were compounded by

differences in training and/or experience using the UTM system 

in a real-world environment, often leaving the operator feeling 

uncertain of the appropriateness of their own actions. As the 

UTM system is developed and its elements achieve more 

automated functions, consideration needs to be given to the way 

the operator’s role will transition while ensuring that user SA is 

maintained. 

All of the challenges outlined in the meta-analysis were 

sustained throughout the maturing of the UTM system and 

should be expected to remain and advance into comparable 

systems if not formally addressed. Looking to future airspace 

operations, with the development of urban air mobility (UAM) 

concepts, that are founded on some of the principles of UTM, 

these findings could be used to inform this future airspace 

exploration and guide further human-system interaction 

research. The obstacles discussed, and their potential negative 

impacts on a human operator’s ability to attain ideal 

performance in these environments, could be alleviated by 

following commonly agreed upon conventions and directions, 

possibly established by a regulating authority. 

ACKNOWLEDGMENT 

The authors would like to thank their fellow researchers 

from the Airspace Operations Laboratory at NASA Ames 

Research Center and other contributing NASA teams for their 

tireless dedication and countless hours of support through five 

years of data collection. The authors would also like to thank the 

six UAS test sites, their partners, and their teams from across the 

United States of America, not only for the invaluable data that 

their participation resulted in, but also for the warm hospitality 

that they showed to our visiting researchers over many weeks 

working on-site together. 

REFERENCES 

[1] J. Robinson III, M. Johnson, J. Jung, P. Kopardekar, T. Prevot, and J. Rios, Unmanned Aerial System Traffic Management 

Concept of Operations (v0.5) (UTM CONOPS), NASA/TM2015-000000, NASA Ames Research Center, Moffett Field, CA, 

2015. 

[2] Federal Aviation Administration, UTM Concept of Operations 

version 2.0 (UTM ConOps 2.0),

https://www.faa.gov/uas/research_development/traffic_manage

ment/media/UTM_ConOps_v2.pdf, FAA, DoT, Washington, 

DC, 2020. 

[3] S. White, NASA’s Air Traffic Control system for drones nearing

completion, Discover Magazine,

https://www.discovermagazine.com/technology/nasas-airtraffic-control-system-for-drones-nearing-completion, Kalmbach

Media Co., August, 2019. 

[4] R. Johnson, Unmanned Aircraft System Traffic Management

(UTM) Project, UTM R and D Workshop, NASA Ames Research

Center, Moffett Field, April 2018. 

[5] J. Homola, L. Martin, M. Cencetti, and A. Aweiss, UAS Traffic 

Management (UTM) Technical Capability Level 3 (TCL3) flight

demonstration: Concept test and results, 38th Digital Avionics 

Systems Conference (DASC), San Diego, CA, 2019. 

[6] Federal Aviation Administration, Certificated Remote Pilots 

including Commercial Operators, https://www.faa.gov/uas/commercial_operators/, FAA, April 30 

2020. 

[7] M. Johnson, J. Jung, J. Rios, J. Mercer, J. Homola, T. Prevot, and 

D. Mulfinger, Flight Test Evaluation of an Unmanned Aircraft 

System Traffic Management (UTM) Concept for Multiple 

Beyond-Visual-Line-of-Sight Operations, Twelfth USA/Europe 

Air Traffic Management Research and Development Seminar 

(ATM2017), Seattle, WA, June 2017. 

[8] P. Kopardekar, Flight Test Evaluation of an Unmanned Aircraft 

System Traffic Management (UTM) Concept for Multiple Beyond-Visual-Line-of-Sight Operations, Twelfth USA/Europe 

Air Traffic Management Research and Development Seminar 

(ATM2017), Seattle, WA, June 2017. 

[9] R. Parasuraman, and C. Miller, Trust and etiquette in high- criticality automated systems, Communications of the ACM, 47, 

iv, P51-55, 2004. 

[10] T. Sheridan, Wiley series in system engineering and 

management: HFES issues in human factors and ergonomics 

series: Vol. 3. Humans and automation: System design and 

research issues. John Wiley & Sons Inc; Human Factors and 

Ergonomics Society, 2002.