NASA/TM—20205003361 

TCL4 UTM (UAS Traffic Management) Nevada 2019 Flight

Tests, Airspace Operations Laboratory (AOL) Report 

Lynne Martin 

NASA Ames Research Center 

Cynthia Wolter 

San Jose State University Foundation 

Kimberly Jobe 

San Jose State University Foundation 

Mariah Manzano 

San Jose State University Foundation 

Stefan Blandin 

San Jose State University Foundation 

Michele Cencetti 

Universities Space Research Association 

Lauren Claudatos 

NASA Ames Research Center 

Joey Mercer 

NASA Ames Research Center 

Jeffrey Homola 

NASA Ames Research Center 

National Aeronautics and 

Space Administration 

Ames Research Center 

Moffett Field, California 

March 2020 
 

 

 

NASA STI Program...in Profile 

Since its founding, NASA has been dedicated 

to the advancement of aeronautics and space

science. The NASA scientific and technical 

information (STI) program plays a key part in 

helping NASA maintain this important role. 

The NASA STI program operates under the

auspices of the Agency Chief Information 

Officer. It collects, organizes, provides for 

archiving, and disseminates NASA’s STI. The

NASA STI program provides access to the

NTRS Registered and its public interface, the

NASA Technical Reports Server, thus

providing one of the largest collections of 

aeronautical and space science STI in the

world. Results are published in both non-NASA 

channels and by NASA in the NASA STI 

Report Series, which includes the following 

report types: 

• TECHNICAL PUBLICATION. Reports

of completed research or a major 

significant phase of research that present

the results of NASA programs and 

include extensive data or theoretical 

analysis. Includes compilations of 

significant scientific and technical data

and information deemed to be of 

continuing reference value. NASA

counterpart of peer-reviewed formal

professional papers but has less stringent

limitations on manuscript length and 

extent of graphic presentations. 

• TECHNICAL MEMORANDUM. 

Scientific and technical findings that are 

preliminary or of specialized interest, e.g., 

quick release reports, working papers, and 

bibliographies that contain minimal

annotation. Does not contain extensive 

analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. 

Collected papers from scientific and 

technical conferences, symposia, 

seminars, or other meetings sponsored 

or co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from

NASA programs, projects, and 

missions, often concerned with subjects

having substantial public interest. 

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material 

pertinent to NASA’s mission. 

Specialized services also include creating 

custom thesauri, building customized 

databases, and organizing and publishing 

research results. 

For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home

page at http://www.sti.nasa.gov 

• E-mail your question via to 

help@sti.nasa.gov 

• Phone the NASA STI Help Desk at

(757) 864-9658 

• Write to: 

NASA STI Information Desk 

Mail Stop 148

NASA Langley Research Center

Hampton, VA 23681-2199 

ii 
 

 

 

 

NASA/TM—20205003361 

TCL4 UTM (UAS Traffic Management) Nevada 2019 Flight 

Tests, Airspace Operations Laboratory (AOL) Report 

Lynne Martin 

NASA Ames Research Center 

Cynthia Wolter 

San Jose State University Foundation 

Kimberly Jobe 

San Jose State University Foundation 

Mariah Manzano 

San Jose State University Foundation 

Stefan Blandin 

San Jose State University Foundation 

Michele Cencetti 

Universities Space Research Association 

Lauren Claudatos 

NASA Ames Research Center 

Joey Mercer 

NASA Ames Research Center 

Jeffrey Homola 

NASA Ames Research Center 

National Aeronautics and 

Space Administration 

Ames Research Center 

Moffett Field, California 

March 2020 

iii 
Acknowledgements 

The authors would like to thank fellow researchers from the Airspace Operations Laboratory at 

NASA Ames Research Center for their contributions to the team while collecting the data presented 

here. The wealth of knowledge obtained through these flight tests would not have been as fruitful 

without their dedication over many long days, over many long months. Both for their efforts in the 

lab and the field, we would like to individually recognize Yasmin Arbab, Latha Balijepalle, Abhay 

Borade, Madison Goodyear, Caterina Grossi, Vimmy Gujral, Nicole Mok, Faisal Omar, Cesar 

Ramirez, Mark Snycerski, and Terence Tyson. 

Trade name and trademarks are used in this report for 

identification only. Their usage does not constitute an 

official endorsement, either expressed or implied, by the 

National Aeronautics and Space Administration. 

Available from: 

NASA STI Program

STI Support Services

Mail Stop 148

NASA Langley Research Center

Hampton, VA 23681-2199 

This report is also available in electronic form at http://www.sti.nasa.gov 

or http://ntrs.nasa.gov/ 

iv 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table of Contents 

List of Figures ............................................................................................................................... vii 

List of Tables................................................................................................................................. viii 

Acronyms and Definitions ............................................................................................................ ix 

Executive Summary ...................................................................................................................... 1 

1. Background ............................................................................................................................... 2 

2. Method....................................................................................................................................... 2 

2.1 Participant Roles and Responsibilities................................................................................ 2 

2.2 Vehicle Characteristics ....................................................................................................... 3 

2.3 Interfaces and Information Displays................................................................................... 3 

2.4 Test Scenarios..................................................................................................................... 4 

2.5 Research Objectives............................................................................................................ 6 

2.6 Data Collection .................................................................................................................. 8 

3. Results ....................................................................................................................................... 10 

3.1 UTM function: USS-to-USS negotiation............................................................................ 10 

3.1.1 Participant and Observer Comments ......................................................................... 11 

3.1.2 Requisite Properties Within UTM for USS-to-USS Negotiation .............................. 12 

3.2 UTM Function: UVR Management.................................................................................... 13 

3.2.1 Participant and Observer Comments ......................................................................... 14 

3.2.2 Requisite Properties Within UTM for UVR Management ........................................ 17 

3.3 UTM Function: Priority Status ........................................................................................... 17 

3.3.1 Participant and Observer Comments ......................................................................... 18 

3.3.2 Requisite Properties Within UTM for Priority Status................................................ 19 

3.4 UTM Function: Remote ID................................................................................................. 19 

3.5 UTM Function: Indication of Conflicts, Conflict Alerting................................................. 20 

3.5.1 Participant and Observer Comments ......................................................................... 21 

3.5.2 Requisite Properties Within UTM for Conflict Alerting ........................................... 22 

3.6 UTM Function: Alerting with a Focus on Alerting for CNS Failures................................ 22 

3.6.1 Participant and Observer Comments ......................................................................... 23 

3.6.2 Requisite Properties Within UTM for Alerting ......................................................... 24 

3.7 Core UTM Function: Volumes........................................................................................... 24 

3.7.1 Participant and Observer Comments ......................................................................... 25 

3.8 Core UTM Function: Indicating Rogue Status................................................................... 25 

3.8.1 Participant and Observer Comments ......................................................................... 26 

3.8.2 Requisite Properties Within UTM for Core UTM Functions.................................... 27 

3.9 Interface Between UTM and its Users................................................................................ 27 

3.10 Interaction with UTM ....................................................................................................... 27 

3.10.1 Interaction with UTM: Communication/ Messaging............................................... 28 

3.10.2 Interaction with UTM: Information Efficiency and Effectiveness.......................... 29 

3.10.3 Interaction with UTM: Procedures .......................................................................... 30 

v 
 

 

 

 

 

 

 

 

 

 

 

 

3.10.4 Interaction with UTM: Situation Awareness........................................................... 30 

3.10.5 Interaction with UTM: Promoting Trust and Confidence in the System................. 31 

3.10.5.1 Promoting Good User Interaction with UTM................................................. 31 

4. Summary and Conclusions........................................................................................................ 32 

5. References ................................................................................................................................. 35 

Appendix A. Crew Member Roles and Responsibilities............................................................... 36 

Appendix B. UTM Architecture.................................................................................................... 37 

Appendix C. Participant and Observer Statement Coding Scheme Based on Research 

Question Topics........................................................................................................ 38 

Appendix D. General Description of Data .................................................................................... 39 

Appendix E. Participant Survey Responses on Negotiations, Priority Status and 

Conflict Alerting....................................................................................................... 40 

Appendix F. Breakout of Negotiation and of UVR Operations.................................................... 43 

vi 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

List of Figures 

Figure 1. Example of the Reno urban environment, the downtown city center .................... 5 

Figure 3. Research question and information topics used for organizing qualitative 

Figure 5. Observation count of items of information received about negotiation 

Figure 2. Example of the Reno suburban environment, a city park ...................................... 5 

data collected .......................................................................................................... 7 

Figure 4. Example of complex urban flights over downtown Reno ..................................... 8 

during flights ......................................................................................................... 10 

Figure 6. Participant opinions on information received about negotiation within UTM....... 12 

Figure 7a. Example of weather UVR during Scenario 1 ....................................................... 13 

Figure 7b. Example of weather UVR during Scenario 2....................................................... 13 

Figure 8. Observation count of items of information received about UVRs within 

flight tests .............................................................................................................. 14 

Figure 9. Participant-reported opinions about the quality of available UVR information .... 15 

Figure 10. Participant ratings of awareness of flight properties and quality of USS 

Figure 11. Observer count of items of information received about priority flights for 

Figure 13. Participant ratings on conflict management: Awareness of conflicts and 

Figure 14. Participant opinions on the the quality of UTM information with ragards to 

interaction during UVR activation ........................................................................ 15 

two data-collection scenarios ................................................................................ 18 

Figure 12. How participants became aware of public safety RID activities.......................... 20 

effectiveness of alerting......................................................................................... 21 

CNS issues ........................................................................................................... 23 

Figure 15. Observer count of items of information shared about ownship status ................. 24 

Figure 16. Example of a flight that has rogue status.............................................................. 25 

Figure 17. Participant opinions on their uneasiness about nearby rogue vehicles and 

clarity of this indication......................................................................................... 26 

Figure 18. List of sources for UVR-cleared messages sent when a UVR is taken down...... 28 

Figure B1. UTM architecture................................................................................................. 37 

Figure D1. Observer counts of uses of UTM information by flight crews............................ 39 

Figure E1. Participants’ desired level of involvement with the UTM negotation process.... 40 

Figure E2. Participant reported timeliness of arrival of priority flight information 

during UVR activation .......................................................................................... 40 

Figure E3. Participant reported properties of messages about plan changes......................... 41 

Figure E4. Alert messages articipants recall seeing on UTM................................................ 41 

Figure E5. Participant reported properties of conflict information........................................ 42 

vii 
 

 

 

 

 

 

 

 

0

0

List of Tables 

Table 1. Crew, Vehicle, and USS Cient Pairings ................................................................... 3 

Table 2. Number of Live and Simulated Flights providing Data for the TCL4-Nevada 

Flight Test .............................................................................................................. 9 

Table A1. Crew Member Roles and Responsibilities............................................................. 36 

Table C1. Participant and Observer Statement Coding Scheme based on Research 

Question Topics...................................................................................................... 38 

Table F1. Breakout of Negotation Operations........................................................................ 43 

Table F2. Breakout of UVR Operations ................................................................................. 43 

viii 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Acronyms and Definitions 

4D...........................................four dimensional 

AOL .......................................Airspace Operations Lab (Human Systems Integration 

Division, NASA Ames Research Center) 

App.........................................(software) application 

ARC .......................................Ames Research Center (NASA) 

BVLOS ..................................beyond visual line of sight 

C2...........................................command and control 

CH..........................................characteristic 

CNS........................................Communication, Navigation, and Surveillance 

GPS ........................................global positioning system 

DMP.......................................Data Management Plan 

FIMS ......................................Flight Information Management System 

GCS........................................Gound Control Station 

GCSO.....................................Ground Control Station operator 

GUFI ......................................globally unique flight identifier 

HSI .........................................human-system interacrion 

ID ...........................................identification 

MOE.......................................measure of effectiveness 

MOP.......................................measure of performance 

NASA.....................................National Aviation and Space Administration 

NIAS ......................................Nevada Institute for Autonomous Systems 

PIC .........................................pilot-in-command 

RC ..........................................radio controlled 

RID.........................................remote identification 

RTB........................................return to base 

RTL........................................return to launch 

SA ..........................................situation awareness 

SDSP......................................Supplemental Data Service Provider 

SOW.......................................statement of work 

sUAS......................................small Unmanned Aerial System 

TCL........................................Technical Capability Level 

TE...........................................test event 

UAS .......................................Unmanned Aircraft System 

UAV.......................................unmanned aerial vehicle 

USS ........................................UAS Service Supplier 

USS Op ..................................USS System Operator 

UTM.......................................UAS Traffic Management 

UVR.......................................UAV volume restriction 

V2V........................................vehicle-to-vehicle 

VO..........................................visual observer 

ix 
 

 

 

 

 

 

 

 

________________________ 

TCL4 UTM (UAS Traffic Management) Nevada 2019 Flight 

Tests, Airspace Operations Laboratory (AOL) Report 

Lynne Martin1, Cynthia Wolter2, Kimberly Jobe2, Mariah Manzano2, Stefan Blandin2, 

Michele Cencetti3, Lauren Claudatos1, Joey Mercer1, and Jeffrey Homola1

Executive Summary 

The Unmanned Aircraft Systems (UAS) Traffic Management (UTM) research project has been 

developing and testing concept ideas for enabling small UAS (sUAS) operations in low-altitude 

airspace (ground to 400 feet). To do this, NASA has organized a series of flight test demonstrations. 

Technical Capability Level-4 (TCL4) flight tests were conducted at a Nevada, USA test site, during 

June 2019 and a Texas test site during August. The Nevada testing resulted in over 300 datacollection flights using eight live rotorcraft, 15 simulated vehicles, involving six flight crews, and 

five UAS Service Suppliers (USS). The TCL4 approach was designed to demonstrate five scenarios 

that set up five diverse sets of UAS events and activities. The Nevada test site focused on three of 

these scenarios: an incoming weather front, a concert event with an incident requiring an emergency 

response, and a scenario where multiple sUAS vehicles experienced Communication, Navigation, 

and Surveillance (CNS) issues. Each of the three scenarios run at the Nevada test site consisted of 

three phases. Each phase was executed three times, creating a total of nine missions per UAS 

scenario. This document presents data collected from participants during the TCL4-Nevada June 

flight test that provides information about how much and how well operators were able to make use 

of UTM functions and information, with the goal of exploring minimum information requirements 

and/or best practices in TCL4 operations. The driving enquiry was: how do UTM tools and features 

support (human) operators leading to safe and effective conduct of large-scale, beyond visual line of 

sight (BVLOS) sUAS operations in “urban canyon” environments? As with the data collected during 

previous similar tests (e.g., TCL3, Martin, et al., 2019), the quality of the UTM information 

exchanged, and the meaningfulness, and therefore usefulness, of this information, were all focal 

points of the questions asked and the data collected. The results aligned with five human-system 

attributes to indicate that UTM provided information that contributed to users’ ability to operate 

safely and effectively within UTM operations, but that information was not always complete and 

was sometimes unclear to the operators. 

1 NASA Ames Research Center, Moffett Field, California. 

2 San Jose State University, Moffett Field, California. 

3 Universities Space Research Association, Moffett Field, California. 

1 
 

 

 

 

 

 

 

 

 

1. Background 

As part of NASA’s Unmanned Aircraft Systems (UAS) Traffic Management (UTM) research effort 

(Kopardekar, et al., 2016), the Technical Capability Level-4 (TCL4) flight tests were intended to 

demonstrate small UAS (sUAS) operations in the most complex nominal environment possible and 

to showcase more complex USS functionality than the previous TCL tests. Two test sites were 

chosen to work on demonstrations: Lonestar, a Texas A&M University organization based in Corpus 

Christi, Texas, and the Nevada Institute for Autonomous Systems (NIAS), based in Las Vegas, 

Nevada, who, for the TCL4 demonstration, operated out of Reno, Nevada. This paper examines the 

NIAS, Reno, Nevada TCL4 flight demonstration only, further information on the Corpus Christi 

flight test can be found in Martin, et al. (2020). 

The UTM concept combines airspace design, flight rules, operational procedures, ground-based 

systems, and vehicle capabilities to enable safe and efficient use of class G airspace by sUAS. When 

extending these operations into airspace over highly-populated and developed areas, the hazards to, 

and risk level of the operations increase. The high density and fast pace of this arena (see FAA, 2018 

or Kopardekar, et al., 2016 for descriptions of the UTM concept), impose more demands on the user 

to fly safely and efficiently and highlights the need for precise maneuvering to avoid frequent 

obstacles. To support operators, UTM information, primarily gained through USSs, but also through 

SDSPs (Supplemental Data Service Providers) and potentially other portals (e.g., remote 

identification [ID] situation awareness tools), needs to be easily usable in a human factors sense— 

that is, it must be clear, concise, consistent, understandable, and straightforward (Krug, 2014). If a 

system is providing users with adequate information, users should report being comfortable with 

their awareness and with taking actions within the system. Approaching the TCL4 demonstration 

from the perspective of the user, with the goal of instructing what the minimum information 

requirements and best practices might be, the driving enquiry was: How do UTM tools and features 

support (human) operators leading to safe and effective conduct of large-scale BVLOS (beyond 

visual line of sight) sUAS operations in “urban canyon” environments? To address this question, 

participants were asked for feedback about the properties of UTM information exchanges. These 

research drivers were overlaid onto the NASA statement of work (SOW) scenarios to develop a set 

of questions presented to UAS and USS operators. 

2. Method 

2.1 Participant Roles and Responsibilities 

Six flight crews took part in the TCL4-Nevada flight tests. Two crews consisted of just one 

individual, while another two crews had approximately six members (Table 1), amounting to 26 

participants who were observed and invited to complete surveys. Primary flight crew positions are 

listed in Appendix A, with many crew members fulfilling more than one role. Participants who 

served as pilot-in-command (PIC) held manned aviation flight qualifications and all flight crew 

participants who had roles related to operating the sUAS had previous UAS operational experience. 

USS software developers had created the USS software clients that were being used. They were 

available to assist flight crews by answering queries and helping resolve issues. Sometimes they 

operated their USS clients for crews during test flights while other developers operated their USS to 

control multiple simulated flights. 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2.2 Vehicle Characteristics 

The vehicles flown during the TCL4-Nevada flight tests were a variety of multi-rotor sUAS 

vehicles, each with varying performance characteristics and endurance limits. There were four 

different models of live aircraft flown (Table 1). These vehicles were able to take-off and land 

vertically in a small area and turn on a point in the air, which is a necessity for urban flying. 

Table 1. Crew, Vehicle, and USS Client Pairings 

Crew Identifier Number of Personnel in Crew sUAS Vehicle Flown USS Client 

GCS A 5+ AR-200 USS B 

GCS C 4 DJI M-600 USS H 

GCS E 1 crew (+ VO + developer all 

the time) 

DJI M-600 USS G 

GCS G 4 DA Nav-X USS H 

GCS H 1 crew (+ VO + developer 

occasionally) 

DJI M210-RTK USS H 

GCS I 6 DA Nav-X USS H 

Sim D 1 e.g., 10 sim vehicles1 USS D 

Sim B 1 e.g., 5 sim vehicles USS B 

Sim F 1 e.g., 8 sim vehicles USS F 

Note: GCS = Ground Control Station; VO = visual observer; AR = Air Robot; DA = Drone America; 

sim = simulated operations; DJI = Da-Jiang Innovation; M = Matrice; RTK = Real time kinematic. 

Three of the live vehicles were controlled either through auto-flight software on a Ground Control 

Station (GCS) or manually by a pilot-in-command (PIC) through a handheld radio controlled (RC) 

unit. The PIC often launched and landed the rotorcraft manually, putting it into autopilot for the en 

route portion of the flight. The other three live vehicles were automatically controlled from launch to 

landing through the GCS’s autoflight software. 

2.3 Interfaces and Information Displays 

Equipment available at each GCS location varied widely across the test site. At most GCSs, several 

displays were available to the flight crews to give them information about their vehicle’s flight. 

Some included displays to show surrounding operations, and/or aspects of the UTM system. For 

example, GCS-H included four displays and a radio that were used by a single crewmember who 

executed all the tasks of a GCS operator (GCSO), a PIC, and a USS operator (USS Op). Standard 

tools on the operator displays were flight planning/execution software, a USS client on two displays, 

and a video feed from the vehicle. At GCS-I, in contrast, the six-person flight crew had two displays 

and a radio. One of those displays showed flight software and UTM information combined, and the 

second display was for UTM planning pre-flight. 

1 Number of vehicles simulated by each “Sim USS” varied across scenarios to give a total of 15 simulated 

vehicles every day across the three USS providers. 

3 
 

 

 

 

 

 

 

 

 

 

The Nevada test site provided additional weather data from stations mounted on buildings and light 

poles locally and offered data from Fortem radar, both of which were sent to the flight test control 

hub where the test director was located, to provide information about the airspace not provided by 

vehicles’ on-board sensors. A third set of information was offered by RelmaTech, which provided 

localized situation awareness for the purposes of remote identification, although crews found this 

information useful as a situation awareness tool in general. 

In the same way that there was a mix of team members and vehicle types, the five partner-built USSs 

also varied. UAS Service Suppliers provide services, via a client, to support the safe and efficient 

use of airspace, which includes communicating between elements of the UTM system, giving the 

user awareness of demand in the airspace to support decision making, and keeping records of flights 

for later inspection or data collection. These partner-built USSs that form the hub of the UTM 

system (Appendix B), interfaced with the FIMS (Flight Information Management System) that was 

hosted at NASA. The tools available within the USSs varied, primarily because each partner 

developed their USS independently to a set of USS-level requirements (Rios, 2017). No standards 

were set regarding user interface design, thus the USS developers were able to present UTM 

information in a wide variety of ways. The USSs were still prototype systems under development 

and had varying functions and features to convey UTM information to crews. To participate in the 

flight tests, all USSs needed to have certain basic capabilities, which they exercised in 

“collaboration” simulations with NASA beforehand (the Collaborative Simulation took place in 

April 2019), but the manner and extent by which the partners met those requirements differed. 

2.4 Test Scenarios 

The test aims were presented in the NASA statement of work as five scenarios encompassing 39 

characteristics (CH) and 16 test events (TE) that, when combined, were designed to portray different 

use cases for sUAS in complex environments (Rios, 2018), e.g., CH1 was a percentage of low 

density operations at 3 or fewer aircraft per 0.2 square miles, and TE1 was operations impacted by a 

dynamic restriction. The scenarios featured 15 simulated vehicles, multiple live vehicles, and 

multiple USSs. These scenarios followed and expanded on most of the characteristics from the 

National Campaign flight tests of 2017 (Martin, et al., 2018) and the TCL3 testing of 2018 (BVLOS 

operations, dynamic re-planning, responses to alerts from the UTM System, and the implementation 

of off-nominal contingency plans) and added characteristic functions like negotiation, UVR (UAS 

volume restriction) placement, and priority status (Martin, et al., 2019). 

NIAS agreed to undertake meeting the criteria for, and flying, three of the five sUAS scenarios from 

the NASA SOW, which included 31 characteristics and 12 events. These scenarios were designed to 

focus on incoming weather (scenario 1), an incident at a large group gathering that required 

emergency response (scenario 2), and large-scale CNS issues (scenario 4). Detailed descriptions of 

the three base scenarios and their corresponding characteristics and events can be found in the TCL4 

statement of work (Rios, 2018). 

With the directions given in the SOW (Rios, 2018), NIAS interpreted the NASA scenario briefs and 

created their own detailed test scenarios to explore the use of UTM in the Reno, Nevada urban 

environment (Brilliant & McGuire, 2019, proprietary, document not available). Two scenarios (1 

and 4) were located directly downtown, in an environment characterized by multi-story, closelypacked buildings (as most downtown city areas are), see Figure 1. The NIAS scenarios defined 

activities for 15 simulated vehicles, five live vehicles, and five USSs, resulting in multi-volume 

flight operations, and dense spacing near obstacles and people. Bases of operations (Ground Control 

4 
 

 

 

 

Stations, GCS) for live flights were located at here-to-fore untested landing zones, including roof 

tops, on city streets, and in empty urban lots. 

Figure 1. Example of the Reno urban environment—the downtown city center. 

Scenario 2, which was run third, was located in a Reno urban park. This environment has few 

buildings but more dynamic obstacles like trees, wildlife, and power lines (Figure 2). NIAS defined 

activities for 15 simulated vehicles, five live vehicles and four USSs in this environment. Ground 

control station locations for live flights were in grassy park areas and open spaces but always close 

to smaller obstacles: trees, fencing, and flag and light poles. 

Figure 2. Example of the Reno suburban environment, a city park. 

Due to the logistic constraints of shutting down some city streets for a number of days, operators at 

the Nevada test site executed each of their three scenarios wholly and consecutively, i.e., completing 

one scenario before moving on to the next. This resulted in an order effect, scenario 1 took place 

during the first week, scenario 4 was at the beginning of the second week, and scenario 2 was at the 

5 
 

 

 

 

 

 

end of the second week. Thus, crews were more familiar with their vehicles, the environment and 

with their USS for scenario 2 than for scenario 1. There were also environmental differences 

between the scenarios. As noted, scenarios 1 and 4 took place downtown, scenario 2 took place in a 

park, which, while still an urban setting, had fewer obstacles that were buildings, and less vehicular 

and pedestrian traffic, but more trees and powerlines. These differences added richness to the data, 

but it should be noted that pilot concerns were different for scenarios 1 and 4 versus scenario 2. 

2.5 Research Objectives 

With the goal of instructing what the minimum information requirements and/or best practices might 

be in TCL4 operations, the driving enquiry was: How do UTM tools and features support (human) 

operators leading to safe and effective conduct of large-scale BVLOS sUAS operations in “urban 

canyon” environments? This enquiry touches on one measure of performance (MOP), one measure 

of effectiveness (MOE) and two USS requirements set out by (SOW, Rios, 2018; MOPS document, 

Rios, 2019). UTM MOP #15 states that the TCL4 research effort should provide feedback on: Pilot 

assessment of UTM information properties (MOP, Rios 2019, sheet 1) and UTM MOE #4 advocates 

investigation of the statement that: UTM allows for common situational awareness of the airspace 

and operations within it to support sUAS operations (MOP, Rios 2019, sheet 2). At a different level 

of enquiry, USS requirements #5 and #6 state that: the USS shall provide human interfaces to 

operators and ensure that the human interfaces are appropriate to support testing activities (SOW, 

Rios, 2018, page 23). These two UTM measures and two USS requirements were re-interpreted from 

the user’s point of view to state that the UTM system needs to: 

• share information with users (through a USS client) 

• provide adequate crew situation awareness (SA) 

For UTM information to be usable, operators need to have enough knowledge in order to: 

• understand what they are seeing both in the airspace and on the GCS 

• be able to respond quickly and appropriately enough to the information when an 

action is needed 

Five main attributes that indicate users’ ability to operate safely and effectively within UTM were 

considered in the data collection at the Nevada test-site: situation awareness (SA), risk perception, 

communication, confidence/ trust in tools, and response quality (Figure 3, green box). 

• The user should have good situation awareness; the ability (based on training) to 

understand the UTM information they have access to, about their operation, and 

about the environment. In turn, these information items are sufficiently usable, 

salient and intuitive. 

• The user should have good risk perception and a high level of safety awareness; the 

ability to differentiate between varying levels of risk of an operation and make 

decisions based on their assessment. The information available should have 

properties that make it support risk management. 

• UTM system-to-user communications should be good; communications (messages, 

notifications, alerts) received by crews are understandable. Communications are 

sent at the right time for users to be able to make use of them. 

• User confidence and trust in UTM should be high; users are able to base their 

decisions and actions on UTM information alone, but are also able to consult 

6 
 

(multiple) other sources if desired. Information is reliable and accurate enough to 

inspire trust. 

• Efficient and effective user responses to UTM information; users are able to act 

effectively through the UTM system. Action options are available and functional. 

These five attributes represent successful user-system interactions from the user’s point of view and 

were used to guide the organization of comments from debrief discussions and observers’ field notes 

(Appendix C). 

In order to promote these types of successful user-to-automation interactions (HSI), the UTM system 

(especially its interfaces, mainly in the form of USS clients) would benefit from having a number of 

properties, some of which are listed in the peach hexagon in Figure 3. Thus, some questions on the 

surveys and during the debriefs were asked from this point of view—how well users thought the 

UTM system and the USS clients worked, and whether the functions needed in both were present. 

Figure 3. Research question and information topics used for organizing qualitative data collected. 

While the key focus of the enquiries described below is the user’s experience, critical to the success 

or failure of this experience is the UTM system and, in particular for TCL4, the USSs as these 

provided the interfaces for users to interact with and view the UTM system. USSs perform many 

functions and a subset of these requires the user to be aware of, and in some cases take part in, the 

functions being executed. Functions that are in this latter category, and were of particular interest in 

TCL4, included priority operations, managing UVRs, conflict avoidance, and dealing with CNS 

issues (grey parallelogram in Figure 3). User-UTM (human-system) interaction is facilitated if 

interfaces have a number of properties, including being easy to understand, providing information at 

the right time (usually quickly enough) and providing all necessary information (peach hexagon, 

Figure 3). Additional features that improve usability are being straightforward or intuitive to operate 

when the user needs to interact with the system and having clear procedures for those actions. These 

7 
 

 

 

 

 

 

 

 

features of USS interfaces, when successfully implemented, enable users to develop situation 

awareness, develop confidence in the tools, and have a calibrated perception of risk, etc. (green 

rectangle, Figure 3). 

2.6 Data Collection 

The TCL4 flight demonstration at Reno, Nevada took place during ten days in June 2019. Prior to 

the test flights, there were ten skill development and practice days run at Reno-Stead Airport during 

May 2019, and four official shakedown (i.e., ‘practice’) flying days, on location, in the Reno city 

center also during May 2019. Flight crews were comprised of individuals from partner organizations 

and, in the case of most teams, traveled to the Reno test site for the duration of the testing. Some of 

these organizations developed USS clients, while other partner organizations focused on drone 

operation and teamed with another partner that provided USS support. There were four USSdeveloper partners who teamed with partner flight crew organizations to take part in the flight tests, 

totaling 14 partners working with the NIAS organization2. 

Over the course of the 10-day Nevada flight test, 28 data-collection runs were flown: eight runs of 

flying for the events in scenario 1, nine for scenario 2, and eleven3 for scenario 4. Across these 28 

scenario runs, 1,222 operations were submitted, of which 318 operations (e.g., Figure 4) took place as 

data-collection flights—247 of these were simulated flights and 71 were live flights (Table 2). The 

approved flight polygons were represented on USS clients and are referred to here as “volumes.” 

Figure 4. Example of complex urban fights over downtown Reno, showing both live and simulated 

operations and their volumes, with more than one USS contributing. Note: Magenta 

polygons are active volumes, operations are rogue in orange polygons. The position of 

airborne UAS are denoted by solid white dots. 

During the ten test days, two teams of researchers from NASA’s Airspace Operations Lab (AOL) 

collected data from participants in the field at the Nevada test site. In the NASA AOL observation 

rooms, a team of UTM developers verified data flowing through the system and a small team of 

AOL researchers remotely collected data on the test scenarios. The main data-collection team was in 

Nevada with the flight crews during the testing. During the first week, a team of four was present, 

2 The City of Nevada also participated in the flight tests as they provided law enforcement and fire crew 

personnel to assist with traffic management but are not included in this count.

3 Not all crews flew vehicles in scenario 4. 

8 
 

 

 

 

 

 

 

 

 

 

and during the second week a team of six were on hand to collect data. Data were collected in a 

number of ways: 

• through observations of the participants during flights 

• through a counting app where observers counted crew interactions with their USS clients 

(see Appendix D for a summary of these data) 

• end-of-day surveys 

• end-of-day group debriefs 

These data-collection methods were included in the DMP (Data Management Plan, Modi, 2019) that 

was constructed to inform and assist test sites with their data-collection process. 

Table 2. Number of Live and Simulated Flights providing Data for the TCL4-Nevada Fight Test 

Live flights 

Simulated flights 

Total operations 

Scenario 1 

23 

43 

66 

Scenario 2 

37 

78 

115 

Scenario 4 

11 

126 

137 

Totals 

71 

247 

318 

Note: Only operations with at least 120 position reports have been included. 

All of these methods solicited feedback on the five areas of interest for these tests (Figure 3). In 

total, 57 crew surveys (an additional seven mission surveys), 11 runs of counting app data, and nine 

end-of-day group debriefs were collected. 

Debriefs. During these end-of-day debriefs, flight crews discussed their experiences interacting with 

the UTM system, specifically their situation awareness and communication. Questions were framed 

in terms of the scenario that had been flown and were approached in terms of the UTM functions 

teams had experienced during that day. 

Counting application. The instances of crews receiving and using UTM information for four UTMspecific types of events that needed to be communicated through the crew were recorded through a 

counting application. Researchers counted how, and when, instances of this type of UTM message 

was communicated4 to the crew. 

Survey. Survey items were generated with the five user-topics (Figure 3, green rectangle) in mind, 

and were presented to the participants in the context of the scenario flown for that day, therefore 

addressing the functions that they had had an opportunity to use. Approximately 75 questions were 

generated across three surveys, but conditions were set so that participants only answered around 25 

at any one time. Most questions used a seven-point rating format, with 7 representing a very positive 

rating and 1 representing a very negative rating. Some questions were multiple choice or openended. Researchers in the field also took notes while they were watching flight tests. 

4 Flight crews may have received the same items of UTM information from multiple sources (e.g., USS client, 

radio, and CO). Researchers coded the primary source of the UTM information in this case. 

9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Operational data. Operational data (e.g., logs and position reports) were provided by, and collected 

from NIAS and their partners, but these data are considered elsewhere in other reports. Following 

the way questions to users were organized, participants’ experiences are first described below 

associated with the UTM information exchange that prompted the experience, and then the ways in 

which that interaction, of user with UTM information, could be improved are outlined. 

3. Results 

3.1 UTM Function: USS-to-USS Negotiation 

USS-to-USS negotiation is a means by which flight operations can occupy the same airspace for a 

short amount of time. To achieve this intersection of airspace, both operations are required to agree 

to the potential to negotiate in advance and must have the same priority level. For TCL4, there were 

two “flavors” of negotiation available: re-planning or intersection. For the first, accepting 

negotiation would require one operation to re-plan its volume; and for the second, an accepted 

negotiation meant both operations could occupy the same volume for a short time as one operation is 

in transit past another. Two scenarios in TCL4-Nevada (scenarios 1 and 2) required USS-to-USS 

negotiation (see SOW, CH 25–31, Rios, 2018), where operation volumes for two UAS were going to 

intersect and the USS client had to coordinate both vehicles safely moving through the 

airspace/volume. 

There were 862 negotiation agreements through the UTM system during the TCL4 Nevada flight 

tests. By far, the majority of negotiation agreements were for intersections (829 [96%]), broadcast 

through over 7,000 messages and only a few re-plan agreements (33 [4%]), broadcast through over 

200 messages (Appendix F). Observers counted crews’ use of information about negotiation from 

both UTM and other sources, recording it in their app. There were 18 counts across two of the 

three scenarios (Figure 5). Counts showed that, in scenario 1, 78% of the negotiation information 

shared through the crew (n = 7) was first received through a UTM source, whereas in scenario 2, 

89% of the shared negotiation information (n = 8) was communicated first through other sources 

(e.g., over the radio). 

scenario 1 - Negotiation scenario 2 - Negotiation 

7 

2 1 

8 

negotiation--UTM source negotiation--UTM source 

negotiation- other source negotiation- other source 

Figure 5. Observation count of items of information received about negotiation during 

flights, n = 18. 

There was some confusion and gray area within the existing protocols for negotiation, which led 

users to question requirements, limitations, and status during negotiations. There were several 

suggestions for adding standard USS settings that would inform how an operation might react to a 

negotiation in the future. Crews reported that their insight into the USS-to-USS negotiation process 

10 


 

 

 

3.2 UTM Function: UVR Management 

A UVR (UAS volume restriction) is one example of a dynamic restriction and a UTM constraint. In 

TCL4, UVRs worked in a way similar to a controlled airspace Temporary Flight Restriction, 

denoting a volume of airspace that has restricted entry criteria for some period of time, for safety 

reasons or because specialists need to use the space. UVR flight-restricted areas were put up in 

scenarios 1 and 2 (see SOW, TE-1, Rios, 2018). In scenario 1, the UVR was to denote an area of 

incoming weather (area with blue border in Figure 7a), shutting down the area to all airborn vehicles 

for safety reasons. In scenario 2, an incident on the ground, requiring response from law enforcement 

and safety personnel, led to a UVR being put in place (Figure 7b), allowing the first responders’ 

UAS unhampered operating space. 

Figure 7a. Example of weather UVR put up during the Reno, Nevada, scenario 1. Weather UVR is 

denoted in royal blue; magenta, cyan and orange volumes are flight volumes. 

Figure 7b. Example of an event UVR put up during the Reno, Nevada, scenario 2. Event UVR is 

denoted in royal blue; orange volume is a flight volume showing rogue; white dots 

are vehicles. 

13 

 

 

 

 

 

reported slightly lower awareness of their vehicle’s parameters. In general, participants requested 

more UVR information, including whether or not a user is allowed in a UVR, clear visibility of 

intersecting volumes for context, better knowledge of the environment (particularly incorporating 

better z-axis information), and that additional details (location, altitude, callsigns) be displayed 

clearly and in one place. 

Very 7

scenario 1 scenario 2 

6 

5 

UTM information aided Reliability of UVR info Trustworthiness of UVR Accuracy of UVR info 

awareness info

1 

Not at 

all 

Airspace Management 

Figure 9. Participant-reported opinions about the quality of available UVR information. 

N-aided awareness = 24, n-reliability = 18, n-trustworthiness = 18, naccuracy = 14, 1-7 scale “not at all” to “very.” 

8 

scenario 1 scenario 2 Very 

high 7 

6 Mean rating 4 

3 

2 

Mean rating 5 

4 

3 

2 

Very 

1 low 

Awareness of location Awareness of volume Awareness of UTM Efficiency of USS Effectiveness of USS 

state interaction interaction 

SA during airspace management (UVR) 

Figure 10. Participant ratings of awareness of flight properties (location, volume and 

state) and the quality of the interaction (efficiency and effectiveness) with 

the USS during UVR activation, n-location = 28, n-volume = 27, n = state 

= 26, n-efficiency = 23, n-effectiveness = 21, 1-7 scale extended to show a 

full standard deviation of responses. 

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

When asked, users wanted to see more information with respect to the UVR event. In particular, 

operators stressed that the USS must send a clear message to all operations involved, detailing 

whether or not they are permitted within the UVR and when they can and cannot enter the UVR, i.e., 

the message that “yes, you are authorized” or “no, you are not authorized” is extremely important. 

This was emphasized by users who received notifications that lacked necessary information and 

therefore, were ineffective. For example, one USS did not show a message or information naming 

the event as a UVR, but rather as a conflict between the operation’s volume and another volume (the 

UVR), which is technically correct but does not include meaningful context to help users take 

appropriate action. In another example, a crewmember described how their vehicle went rogue on 

entering a UVR but the UVR boundary was not displayed on the operator’s interface to explain why 

their vehicle was rogue. 

Crews suggested a variety of interface elements through which this information could be displayed, 

including labels, pop-up windows, and messages. The most common request was a visualization of 

UVR volumes, with clear labeling (see Arbab, 2019) for a discussion of interfaces and their 

requirements). One suggestion was that there should be a standard display for UVRs, so that no 

matter which USS an operator looks at, the UVR information can be found in the same way. 

Crews did acknowledge there could be data saturation from UTM/their USS due to the amount of 

information that could potentially be presented with a UVR. Crews discussed management of 

information to prevent overloading the user. Participants said they want to be informed about 

rogue aircraft in a UVR, but would not need to know about priority aircraft that are allowed into 

that volume. 

Pilots emphasized that knowing about the UVR itself was very important, especially if the UVR was 

going to encompass or overlap their volume. The reasoning for needing this level of information is 

that when a UVR is put in place, an operator needs to decide their course of action. They need to 

consider whether they should go around the UVR, or loiter until it expires, and whether they have 

enough fuel to take either option. One issue that made this decision impossible for some crews was 

that they did not have deviation capabilities in their USS, meaning they could not re-plan in response 

to the UVR. Another issue that made managing UVRs difficult was USSs that do not publish the 

operational globally unique flight identifier (GUFI), as without the GUFI information, the USS that 

was publishing the UVR could not give access to the operation with “no GUFI.” These crews 

emphasized that USSs need to have functionality that enables users to avoid the UVR or divert. One 

crew noted the broader need for more functionality in their USS, especially the ability to change 

their operation plan if something unexpected occurred while they were airborne and they called out a 

UVR event as an example. As they were unable to change their flight plans while in flight, they were 

unable to react to the UVR. However, crews did note that for UVRs, their knowledge could be 

localized. If the UVR covers a large area they do not need to see the whole space, just the part of the 

UVR that encompasses or touches their volumes and other volumes that are close by, and 

information on whether it impacts them. This implies that there needs to be a way to show that a 

very large UVR is in place without showing the entire UVR. 

While public safety officers noted they would want to see all the operations within a UVR, both nonpriority crews and public safety officers agreed that non-priority users would not need to see 

complete information about other users, including access permissions (allowed/ not allowed), only 

their own. 

16 
 

 

 

 

 

 

 

 

 

3.2.2 Requisite Properties Within UTM for UVR Management 

Although crews used the majority of the UVR information they received, observers noted that UVR 

interfaces did not give the user enough information. There was a lack of clear visual notifications 

and a lack of written information (via messages). Despite this, researchers observed examples of 

crews sharing awareness of the parameters that drive UVRs within UTM and awareness of their 

operation’s location and status relative to the UVR. However, they also found examples of crews 

who lacked awareness of UVR parameters and how to react to a UVR becoming live around their 

volume. Crew feedback was focused on situation awareness (SA) – and the need to be provided with 

enough information to understand why the UVR had been established, the nature of the restriction, 

and how they, as the user, can adhere to the new restriction. Unless they were first responders, crews 

did not want to know who stayed in the UVR. Taken together, these comments suggest that: 

• More information than is currently available through UTM needs to be provided to users 

and the requirement to provide these items needs to be clearly stated. 

– Ordinary users need a clear permitted/not permitted in the UVR message. 

– All users need a clear, standardized depiction of the UVR and its state (e.g., proposed, 

active, etc.). 

– Priority users need awareness of other operations in the UVR. 

These properties will improve human-system communication, support crew situation awareness, 

increase user trust and help users to operate and act as UTM (and other users) expects. 

• Provide UTM information and messages that are more complete and that clearly define the: 

– UVR and its parameters. 

– operator’s permission status with respect to the UVR. 

• USS functionality that enables crews to respond to a UVR. 

These properties will support crew situation awareness and increase user trust and help users to 

operate and act as UTM (and other users) expects. 

3.3 UTM Function: Priority Status 

One property that was touched upon in both negotiation and UVR discussions was a user’s UTM 

status. Within UTM, users can gain a priority status, which elevates their ability to access airspace. 

There are two types of priority status: for a role that requires access, typically public safety or first 

response, and for operations experiencing an emergency, e.g., due to a low battery, mechanical fault, 

or issues with CNS. The first responder priority status may be designated when an operation submits 

a first response mission, and it will remain designated for the whole operation. The emergency 

priority status is a temporary designation, issued only when an operation declares an emergency, and 

lasts until the vehicle can reach a safe landing location. Two scenarios in TCL4-Nevada (2 and 4) 

involved priority vehicles; in scenario 2, priority was granted for special access to the UVR and in 

scenario 4 priority was granted due to reduced vehicle capability (see SOW, CH-8, Rios, 2018). 

Observers counted how often crews used information they received about priority operations from 

UTM and other sources. There were 18 counts across the data-collection period. In both scenario 1 

and scenario 2, priority status information was received through UTM half of the time and through 

other methods (e.g., radio) the other half of the time (n = 8) (Figure 11). In scenario 4, priority status 

information was always received through a source other than UTM (n = 10). Participant reports 

17 

 

 

 

 

 

 

 

 

 

 

 

 

priority status was not clear to all of the TCL4 participants. In addition, crews were unsure whether a 

USS could distinguish priority status from nominal operations. During debriefs, crews debated how 

to simplify and clarify priority status guidelines, stressing that the rules need to make it easy for 

users to determine their own status and those of other vehicles. Easy recognition would allow 

operators to make decisions about their actions, e.g., whether to proceed into/ within the UVR area 

or to vacate that space. 

There was also some uncertainty when crews needed to give way to other vehicles that had priority 

status. First, it was not clear to them which vehicles had priority status and were exempt from the 

RTL rule when a UVR came into place. Second, users were not always certain when a vehicle that 

had an emergency-priority was flying through or close to their volume, and they commented that 

they needed clear and direct messages to inform them so they could respond appropriately. 

3.3.2 Requisite Properties Within UTM for Priority Status 

Crew feedback centered around wanting clarification of the guidelines that determine priority status 

and also improving the quality of priority messages. 

• More information than is currently available needs to be provided to users: 

– Ordinary (non-priority) users need clear messages about when a priority 

user will be in or near their volumes. 

– Ordinary users need clear guidance (possibly notifications) about priority 

user actions that are acceptable as a priority flight but not as an nominal user. 

– A clear means of indicating a vehicle has priority status. 

These properties will support crew situation awareness and understanding, raise user confidence in 

the UTM system, improve operator communication and increase user trust. 

• Clear UTM guidance needs to be provided that details: 

– How priority users are determined. 

– When priority status applies. 

– Tte UTM guidelines that priority users are exempt from and those which they 

are not exempt from. 

These properties will support crew situation awareness, increase user trust in UTM and improve 

compliance with UTM guidelines. 

3.4 UTM Function: Remote ID 

Remote identification (RID) is the process by which a third party makes a query for key identifying 

details or information about an operation. There are a number of ways this can be done, either 

through the UTM system or through independent software, such as an SDSP. The first steps toward a 

RID function were offered both in and outside of UTM at the Nevada flight tests. Small radio 

broadcasting units enabled the RelmaTech company to complete UAS position locating through 

vehicle-to-vehicle (V2V), and display the results on an application that was not connected to UTM. 

Operators at the Nevada test site also tried running identification queries through the UTM system, 

testing both methods in scenarios 2 and 4 of TCL4 (see SOW, CH 34-37, Rios, 2018). 

19 


 

 

 

 

 

 

 

they need to see the conflicts that are close to their vehicle, and that outside that local threshold/ 

radius, the information is less important. 

Crews emphasized the importance of accurate messaging and position reporting so they could be 

sure when other vehicles were straying into their volumes and when they were not. Although this 

report does not address technical issues with UTM, within the function of conflict reporting there 

was a technical issue that affected whether crews were made aware of conflicts. Differences in the 

ways vehicles reported their altitudes were highlighted by more than one crew. This resulted in both 

false positive conflict alerts and false negatives where alert notifications were not sent for actual 

conflicts. While this is an issue on its own for the UTM system and single-vehicle conformance 

reporting, vehicles that are reporting their altitude using uncalibrated or different base measurements 

negate the usefulness of conflict detection and alerting. Crews noted that as long as they lack 

confidence that other vehicles are flying at the altitude presented on the user interface, they cannot 

rely on UTM-based conflict alerts. Standardization of base measurement and accuracy of position 

reporting is key for conflict representation. 

3.5.2 Requisite Properties Within UTM for Conflict Alerting 

Crews emphasized their preference for a visual representation of conflict situations, with displays 

providing up-to-the-second information. However, they reported a low level of confidence in the 

alerting currently available due to concerns about technical issues. 

• Users would welcome additional information about ownship-other conflicts: 

– users favor a visual representation of conflicts. 

– optionally limit conflict representation to a local area. 

– users need clear messages about the time and location of conflicts. 

These properties will support crew situation awareness and decision making, raise user confidence in 

the UTM system, and improve operator-USS communication. 

• Demonstrably accurate (altitude) measurements need to be provided, which will primarily 

increase user trust in UTM and improve alignment with UTM guidelines by supporting 

accurate crew situation awareness. 

3.6 UTM Function: Alerting with a Focus on Alerting for CNS Failures 

In addition to conflicts and being outside of volume parameters, alerts were sent through the USSs 

for a number of other UTM states, e.g., to warn of UVRs and, for some USS, to warn if a USS-USS 

negotiation resulted in an operation re-plan, and to warn crews of vehicles that were experiencing 

simulated CNS issues. All three scenarios had at least one vehicle that simulated CNS issues, but 

scenario 4 included a large-scale simulation of CNS failure that affected many operations (see SOW, 

TE-11 to 13, Rios, 2018). 

UTM alert messages were shown across a number of displays. One crew had UTM information 

integrated into the PIC flight-planning interface, enabling the pilot to receive and see UTM messages 

while operating the vehicle. Observers commented that alert messages were clear and effective on 

some of the USS interfaces. In general, crews who used their USS and other information to track the 

location of their aircraft at all times reported their SA was good. However, observers noted that 

crews used multiple sources of information. For example, when the pilot received UTM messages 

about a rogue state due to loss of command and control (C2), the pilot cross-checked this 

22 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Some crews suggested it would be useful to receive alerting about a broader set of conditions than are 

currently available through UTM. In addition to weather alerts, it would be useful to receive messages 

about global positioning system (GPS) degradation and cellular networks that are giving intermittent 

coverage. Other crews suggested that such network outages or issues should be posted on a SDSP 

application, rather than as a UTM alert, but there was general agreement that good quality CNS 

information should be accessible to crews. One pilot noted that, although the alerts were useful, the 

repeated alerting of the same message was distracting. The pilot would like the ability to turn off 

certain alerts so that crews could manage the amount of information they received from UTM. It was 

suggested that USSs increase the number of settings, or customization options, allowing more crew 

choice in the alerts they receive. 

3.6.2 Requisite Properties Within UTM for Alerting 

Implementation of UTM notices should consider the following qualities: 

• Information and alert notices need to convey their message in a clear and concise way. 

• Information and alert notices need to clearly state their focus and cause for being sent. 

• Information and alert notices need to persist long enough for the operator to see them. 

• Alert notices need to have a design that draws attention. 

Direction on which notices need to be alerts and which notices are for crew information. These 

properties will support crew situation awareness and decision making and raise user confidence in 

the UTM system. 

3.7 Core UTM Function: Volumes 

One of the core functions of UTM is to check operational volumes that crews have submitted against 

FIMS rules and other constraints. Operators define and submit 4D volumes through their USS, and 

then receive a notification of whether the system has accepted their proposal. They activate the volume 

as they launch. All USSs used in the Nevada test flights showed the own-operation volumes that had 

been activated on a map, but not all showed information for other operations in a similar way. 

AOL Observers counted the uses of information about own-ship operations from UTM and other 

sources (Figure 15). There were 115 counts across the three scenarios. In scenario 1, information about 

own-ship operations was received through UTM 77% of the time (n = 33). In scenario 2, information 

about own-ship operations was received through UTM just a little more often, at 81% of the time (n = 

39), and in scenario 4, information about own-ship operations was received through UTM 71% of the 

time (n = 17). 

scenario 1 - own scenario 2 - own scenario 4 - own 

op op op 

33 

10 

own op--UTM source 

own op--other source 

39 

9 

own op--UTM source 

own op--other source 

17 

7 

own op--UTM source 

own op--other source 

Figure 15. Observer count of items of information shared about ownship status, n = 115. 

24 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3.8.2 Requisite Properties Within UTM for Core UTM Functions 

Crews found UTM volume status information useful. However, they did not fully trust it, and this 

had a valid foundation. At times, the input information was incorrect, which led to the volume status, 

at times, being incorrect. To increase user trust: 

• Information about own-ship volumes should meet the criteria that are applied to all USS: 

– actions on own-ship volumes should be available throughout the operation’s cycle 

– information about own-ship volumes is consistently presented 

These properties will support crew situation awareness and decision making, and raise user 

confidence in the UTM system. 

• The way USSs display rogue-state messages needs to be clear, consistent, and concise: 

– criteria determining when a rogue state message is sent should be common 

across USSs (improve consistency) 

– improve accuracy of rogue-state messages 

These properties will support crew situation awareness and compliance with rogue messages, 

assisting crew reactions and increasing trust in the UTM system. 

3.9 Interface Between UTM and its Users 

UTM has many interfaces but the primary interfaces of interest in the TCL4-Nevada flight tests were 

five USS clients designed and built by five different USS developer organizations which were used 

by the live and simulated flight crews. Of particular interest were the three USS clients used by the 

five live flight crews. The USS client interface is a crucial bridge in the UTM system between users 

and automation. The analysis of the users’ opinions of the USS client interfaces available in the 

Nevada flight tests can be found in a separate report by Arbab (2019). 

3.10 Interaction with UTM 

The core and critical functions of the UTM system work in two ways: 1) by exchanging information 

somewhat autonomously between the elements of the UTM network, e.g., USS-to-USS; and 2) by 

providing information to users (in this report, to flight crews) who need to be aware of and act on this 

information. This crew-USS interaction (or human-system interaction–HSI)) affects users’ perceptions 

of the system and the way they approach being part of and interacting with that system. Four attributes 

of user affect were probed in surveys and debriefs: SA, trust, knowledge gained, and communications. 

Although the UTM system is designed to be autonomous in many respects, users were required to 

interact with the system via their USS, to submit volumes and manage operations, and then to 

monitor their interface for feedback and updates. While there are many properties of interface design 

that have been researched (see work by Nielsen & Molich (1990) and Norman (2013) for example) 

essentially users should be provided the information they need to accomplish their work by their 

tools. To complete their tasks safely and efficiently, users need USS interfaces to support: 1) their 

understanding of what is occurring in the UTM system and in the physical airspace, and 2) their 

ability to interact with thte user interface to change the course of those events if needed. Therefore, 

users are looking for the UTM system to provide them with appropriate information that they can 

trust, supporting them to build good SA of the environment, and that they can effectively 

communicate with the system to change its actions. Users were queried about whether the UTM 

system provided them with these four attributes that they needed. 

27 
 

 

 

 

 

 

 

 

 

3.10.1 Interaction with UTM: Communication/Messaging 

The starting point for successful interaction with UTM is for the system to communicate well with the 

user. Observers noted that many of the messages received reflected good effective communication 

between UTM/USS and the user. For instance, flight crews appeared to have a solid understanding of 

the next steps and actions to take once they received messages from the system. One example of UTM 

messaging was the “all clear” message that crews received after a UVR was taken down and operations 

could resume in that area. Fifty-eight percent of respondents saw the “UVR-cleared” message during 

either scenario 1 or scenario 2 during the Nevada flight tests (n = 14) (Figure 18). A third were 

informed by word of mouth, while a very small number were unaware when the UVR was cleared. 

Didn’t see anything/Don’t know 

No USS display 

Radio transmission 

Informed by teammates 

Told another way 

Yes, there was a message 

Yes, there was a message and/ or an alert 

Yes, there was a change on the map 

Source of UVR-cleared information 

scenario 2 scenario 1 

0 2 4 6 8 

Count 

Figure 18. List of sources for UVR-cleared messages sent when a UVR is taken down, n = 24. 

There were numerous reports from observers that UTM messages and communications were timely 

and effective. Observers described crews using the messages they received to confirm their actions 

and to make decisions. They also noted that crews understood the messages they received and knew 

how these mapped on to actions they needed to take. Observers estimated that USS message displays 

were clear and effective. 

Communication issues resulted from the user feeling that a message was either not delivered 

effectively or that messages lacked necessary information. In one instance, a user did not know that 

they were receiving an alert until someone else pointed it out. Another alerting issue pertains to the 

actual timing of the system sending messages. In some cases, the user received the message over the 

radio before receiving it through the USS. Ideally, alerting messages would be received in a timely 

manner to allow for users to react. There were also instances of messages that lacked necessary 

information and therefore, were ineffective. For instance, the implementation of a UVR must trigger 

a USS to send a clear message to all operations in the area as to whether or not they are permitted 

within the UVR. Overall, users need ample information in order to be fully knowledgeable of a 

situation before re-planning. Missing information in a message may result in a delay that could 

prevent a successful operation or re-plan. 

28 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

As indicated above, some crews were less trusting of UTM, relying on visual feedback and radio 

communications rather than USS messages. Observers noted that those crews who did not have other 

operations displayed on their USS used as many other means as they could to confirm their locations, 

including Slack (Slack Technologies, 2019) and voice comms. These crews with limited “windows” 

into UTM were assessed by observers as trusting the UTM system less than those who had more 

complete displays. For those crews whose USS was only able to show them their own operations, the 

RelmaTech app provided a situation awareness display that showed nearby operations. 

Key USS communications are those that provide information about basic status and airspace changes. 

As noted above, users did not feel that the status messages were clear enough. The presentation of 

status messages should be salient, the meaning of the message should be unmistakable, and the rules 

that govern status should be standardized and consistent, and provided in guidelines. 

3.10.2 Interaction with UTM: Information Efficiency and Effectiveness 

In order to be usable information for the UTM user, data provided by the UTM system needs to be 

useful, accurate, reliable, and conveyed in a timely manner. Crews discussed the degree to which the 

USSs provided information to their team and how this could be improved. 

In terms of information that crews would like to receive, crews noted that, when they are flying live 

operations, they would like as much information about their own flight, other flights and UTM 

activities as possible. They stressed that knowing where other aircraft are is a critical piece of 

information, especially if UTM predicts that other operations are going to be close to their own or, in 

particular, intersecting at some point, as this information was not available from some USSs. Crews 

discussed that sharing information benefits everyone. If one crew broadcasts their situation to all 

those affected that they have changed their plans, this would allow other crews to amend their plans 

so that they are minimally inconvenienced by the changes. 

There was also a debate about the number of displays and devices a crew had to monitor. Some 

pilots commented that having to assimilate and process information from three or more sources was 

workload intensive and made the information they received more difficult to act upon. They 

suggested that combining all the information into one location would streamline decision making. 

They advocated that the GCSO should receive all USS messages and that the messages should have 

a link back to the information that triggered them. 

At the same time, other crews noted that some non-critical but very useful information would be 

well-placed in a different location, such as in an SDSP. Weather information is one example of this. 

Crews noted there is a lack of weather information between the ground and 400 feet, and that an 

application or site that could collect and present weather information at these altitudes would be 

extremely useful. 

There were also some items of information that users found difficult to interpret or use. For example, 

crews reported confusion when activities in UTM did not go as planned. When a crew’s submission 

was rejected, the crew was not provided an explanation for the rejection. They commented that it 

took too much work to try to find the reason. Observers noted that flight crews needed assistance 

from the USSs developer to understand their flight status and other messages. This suggests that the 

USS interface needs to include more explanatory information so that crews understand why UTM 

events are happening. 

29 
 

 

 

 

 

 

 

 

 

 

 

However, other participants noted the possibility that there could be information overload if a USS 

presents all messages and interactions within UTM. Crews discussed that although they would like a 

lot of information, they would not want it all to be visible at the same time. In another discussion, 

crews stated that they had a variety of tasks and could not watch the USS all the time. Observers 

noted that this was the case and that, on a couple of occasions, UTM messages arrived and crews 

were not aware of them as they were doing other things. In debriefs, crews discussed a number of 

ways the USS interface could be enhanced to make more information accessible (see Arbab, 2019). 

During discussions, crews agreed that information about other close-by operations should be 

considered critical, high-priority information. 

3.10.3 Interaction with UTM: Procedures 

To use information effectively, users need strategies, which can be defined as procedures if the 

action sequences have to be made often. Because the USS clients had many new elements, there 

were few formally-defined procedures for operating with UTM during the flight tests in Nevada. 

Crews drew often on pilot procedures from manned aviation, e.g., one GCSO counted down the 

UAS phase change aloud, so the whole crew knew about it. Crews found some of these newlyformed procedures to be effective when they tried them, while others were ineffective. The focus of 

the current paper is on UTM information exchange and does not explore procedure development for 

UTM but consensus should be reached among users about how best to implement often used task 

sequences and which should be standardized as formal procedures. 

3.10.4 Interaction with UTM: Situation Awareness 

Humans use the information they receive to build situation awareness (SA) of events and their 

circumstances. In general, people find it easier to build SA if the information they are using is clear 

and formatted in a way that means they do not have to work to interpret or manipulate the 

information before they can use it. 

Observers recorded instances of crews using their USS to establish situation awareness and instances 

where they did not. Observers reported that those crews who used their USS, did so regularly, to see 

UTM information and check to confirm their UTM state. For one USS that showed other operations 

using the same USS company, crews also checked positions of other vehicles. The observer 

estimated that crews obtained useful situation awareness from these clients. For instance, in one run, 

observers noted that two crews saw through their USS displays that their vehicles were coming 

closer within their volumes than they (the crews) were comfortable with. Using their USS displays, 

crews resolved the issue by discussing their intentions with each other and worked out a series of 

steps where one vehicle loitered while the second maneuvered around the first. 

Some crew-members actively supported building their awareness by following up on information 

they saw on their USS via radio or person-to-person. Observers noted instances of the PIC talking to 

their USS operator to make sure they were fully aware of the situation. The added advantage of these 

discussions was that they ensured everyone impacted by the information was aware of it. In addition 

to basic information, crews discussed the importance to SA of having context to support UTM 

messages. For example, in cases of conflict, the pilot is able to make better decisions if they can 

know from which direction the conflict is going to occur and what distances are involved. 

Observers noted that there were a couple of missing items of information that hampered crew 

situation awareness. Missing information was particularly detrimental during periods crews could 

only see their own-ship operations and could not see other nearby operations. 

30 

 

 

 

 

4. Summary and Conclusions 

UTM Technology Capability Level-4 (TCL4) flight tests were completed in Reno, Nevada during 

June of 2019. Pilot participants flew 318 operations, of which 71 were live flights, to demonstrate 

sequences of events within three sUAS scenarios. The qualitative data discussed above were 

collected by on-site AOL researchers, and consisted of end-of-day debriefs, end-of-day surveys, 

observer notes, and observer event counts. 

From our research question, observing the Nevada TCL4 flight test demonstrated that crews were 

supported by UTM functions when flying in urban environments, however, they were not fully 

supported as in some cases, the user interfaces did not provide clear information to the users. User 

feedback suggests that they would like to act appropriately when negotiating with other USSs, when 

responding to UVRs or conflicts, and when interacting with/as a priority operation, but they lacked 

clear guidance about the bounds of appropriate actions for each situation. Crews would like to know 

the cause of events, as the behavior of the system would then become more predictable, and they, as 

users, would also be better equipped to act as expected. For example, when crews understood the 

system logic and features available to them as a priority operation with simulated failures, they 

thought it useful to be able to elevate their priority, construct a new volume to accommodate a direct 

return to base (RTB), and maintain their conforming state. An opposing example is when one crew 

initiated a RTB without a new volume, then encountered rogue messages that they did not expect, 

did not know the cause of, nor know what the best response would be. In sum, UTM has the 

potential to foster common situational awareness of the airspace and operations within it to support 

sUAS operations (MOP, Rios 2019, sheet 2, MOE 4) but requires improvement to meet that. More 

insight or visibility into the processes, and how all elements of the system behave, could level the 

playing field of expectations and increase overall user situation awareness and system efficiency of 

the system beyond current day. 

One possible method to aid users’ awareness is through UTM messages and notifications, keeping a 

balance between too much and too little information. The information should be rich enough to 

provide context, be user-centric and relevant to the own-operation, arrive at the right time and with 

the right frequency. Some information elements specifically mentioned were to include the reason 

for rejected negotiations, permissions for being within a UVR, explanation for state changes and 

within conflict messages, and give the timing and nature of those conflicts. Crews would like these 

messages to be clear and concise, and if the message is high priority, be salient enough to draw their 

attention, and persistent enough to be seen if their attention is not actively on that display. They 

discussed having visual representations displayed for volumes, UVRs, and detected conflicts, but 

stated that, as with text notifications, these should also be relevant to their own operation. Observers 

noted that crews were sometimes saturated with too much information, and crews stated that it 

would help to limit the information on their display to what impacts them, so they don’t have to read 

and filter while their vehicle is on a mission. Some crew members suggested allowing users to 

customize options for notifications, messages, and alerts. 

Beyond effectively relaying relevant information to the crews and supporting their understanding of 

what action is appropriate once that information is received, crews wondered if automation could 

offer solutions to issues. For instance, if the known proper response to a UVR being erected over 

non-permitted active operations is for those operations to RTB, then there could possibly be a 

prompt delivered to the user that, once clicked, draws a new deconflicted volume. Stated concerns 

about the ability to “game” the system to achieve a desired outcome, the possibility for many 

cascading negotiations or unforeseen future conflicts from negotiations could also possibly be 

32 
 

 

 

mitigated through additional automation, user customization, and/or more education for the user on 

appropriate actions to take in different situations. 

For human-system interaction to be successful, the user needs to have an understanding of the 

way a system works in order to use this knowledge to develop current and future situation 

awareness—in Endsley’s terms, comprehend the meaning of the situation and project into the 

future, (Endsley, 1995). For the UTM system, crew knowledge is centered around two things: a 

general understanding of the concept and understanding the guidelines that govern the way UTM 

functions work in terms of HSI. Feedback from crews in the Nevada flight test suggested that the 

guidelines available for the UTM functions exercised in the flight test were not detailed enough 

for them to be able to successfully use and follow in the test site’s operational setting. Overall, 

crews requested more guidelines about UTM conditions, UTM status, and UTM restrictions. The 

conditions crews were concerned about were those associated with USS-to-USS negotiation, their 

status concerns were linked to when they or another local user were assigned priority, and their 

restriction concerns were for when UVR were in place. Discussion suggested more information in 

the form of guidelines about the implications of selecting or working under a set of conditions 

would assist users, for example, what the implications are if you agree your operation will 

negotiate. Also, more guidelines about orders of priority status and the implications of being a 

higher or lower priority than a nearby user, and more information about the options available 

when a UVR activates. 

Data collected from flight crews aligned with five attributes (Figure 3) and indicated that UTM did 

provide information that contributed to users’ ability to operate safely and effectively within UTM, 

but that information was not complete. 

• Crews reported they had good situation awareness of their own-ship. As noted 

above, they did not always fully understand the UTM information they had access 

to, and had questions about their operation and the environment. Some of the items 

of information items requested were not sufficiently usable, salient or intuitive. 

• Crews did have a high level of safety awareness and were very tuned to risk 

perception and management. While the risks of urban canyon TCL4 flying were 

clear to crews, they were less sure of, and discussed in detail, future risk 

situations where flights have less oversight and preparation, e.g., a real safety 

UVR where crews have to make immediate decisions about whether to return to 

launch and how to exit the space. They emphasized the need for more clarity in 

guidelines for these situations and information to be made available that is 

easily usable for risk management. 

• UTM system-to-user communications were possibly better than users were 

aware of. Few messages were alerted, either visually or aurally (although the 

four USS varied greatly in their interfaces), with the result that sometimes crews 

did not see messages. Most communications (messages, notifications, alerts) 

received by crews were understandable but, as described above, there were 

occasions when the notification crews received was not the one they expected. 

• Users were not required to base their decisions and actions on UTM information 

alone in the TCL4 test, there were multiple sources available for them to crosscheck information. Observers noted a good amount of crosschecking occurring 

across the test suggesting that confidence and trust in UTM was being established. 

33 
• Crews felt they responded efficiently and effectively to UTM information that required 

them to take action. However, often they chose to act outside of UTM to solve the problem 

using manual skills, but it is not clear whether users did this because they could not use 

UTM or through habit. 

The TCL4 flight tests successfully demonstrated flying UAS through an urban canyon and other 

highly-populated areas. Qualitative data collected showed the usefulness of the UTM system and 

the need for information exchange for users of the system. Survey and debrief responses 

underlined crews’ desire to be informed about operations other than their own and their need for 

clear procedural guidelines and clear displays of information. The findings presented above 

complement data previously collected for TCL3 (Spring, 2018), TCL2 National Campaign 

(Spring, 2017), and TCL2 (Reno, 2016). The increased functionality of UTM markedly increased 

its usefulness over previous flight tests, but also identified that some of these functions would 

benefit from more consideration. 

34 
 

 

 

 

 

 

 

 

5. References 

Arbab, Y. (2019). Display design recommendations for USS Client interface based on Technical 

Capability Level 4 (TCL4) Flight Tests. Unpublished Master’s Project, San Jose State 

University, San Jose, CA. 

Brilliant, M., & McGuire, C. (2019). Nevada scenario files. NIAS, proprietary working document, 

not for release. 

Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems, Human Factors, 

37, 32–64. 

Federal Aviation Administration (2018). Unmanned Aircraft System (UAS) Traffic Management 

(UTM), Concept of Operations v1.0, Washington, DC, May 2018. 

Kopardekar, P., Rios, J., Prevot, T., Johnson, M., Jung, J., Robinson III, J.E. (2016). Unmanned 

Aircraft System Traffic Management (UTM) Concept of Operations, AIAA Aviation, 

Washington, DC, 13–17 June. 

Krug, S. (2014). Don’t make me think, revisited, New Riders, USA. 

Martin, L., Wolter, C., Gomez, A., & Mercer, J. (2018). TCL2 National Campaign Human Factors 

Brief, NASA/TM-2018-219901, NASA Ames Research Center, Moffett Field, CA. 

Martin, L., Wolter, C., Jobe, K., Homola, J., Cencetti, M., Dao, Q. & Mercer, J. (2019). TCL3 UTM 

(UAS Traffic Management) Flight Tests, Airpace Operations Laboratory (AOL) Report, 

Technical Memorandum NASA/TM-2019-220347, NASA Ames Research Center, Moffett 

Field, CA. 

Martin, L., Wolter, C., Jobe, K., Goodyear, M., Manzano, M., Cencetti, M., Mercer, J. & Homola, J. 

(2020). TCL4 UTM (UAS Traffic Management) Texas 2019 Flight Tests, Airspace Operations 

Laboratory (AOL) Report. Technical Memorandum NASA/TM-2020-220516, NASA Ames 

Research Center, Moffett Field, CA. 

Miller, C. & Parasuraman, R. (2007). Designing for flexible interaction between humans and 

automation: Delegation interfaces for supervisory control, Human Factors, 49, pp. 57–75. 

Modi, H. (2019). Data Management Plan, rev 2, NASA ARC, CA, https://github.com/nasa/utm-docs. 

Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces, SIGCHI conference on 

Human Factors in Computing Systems CHI'90, Seattle, WA, 1–5 April, pp. 249–256. 

Norman, D. (2013). The design of everyday things, Basic Books, New York. 

Rios, J. (2017). USS specification requirements, NASA Ames Research Center, working document, 

not for wide release. 

Rios, J. (2018). UTM TCL4 Statement of Work, NASA NASA Ames Research Center. 

Rios, J. (2019). UTM MOPS and MOE, NASA NASA Ames Research Center, not for release. 

Slack Technologies (2019). Slack, https://slack.com, San Francisco, CA. 

35 
 

 

 

 

 

 

 

 

 

 

 

Appendix A 

Table A1. Crew Member Roles and Responsibilities 

Crew Member Role Crew Mmber Responsibilities 

Pilot-in-command (PIC) Served as the main pilot for the vehicle 

GCS operator (GCSO) Worked the vehicle’s flight planning and flight execution software 

USS operator (USS Op) Monitored and interacted with USS displays (& NASA) 

Hardware and software 

flight engineers Supported specific technical aspects of the vehicle 

Visual observers (VO) Safety monitors who provided visual contact with the vehicles at all times 

Additional Roles Responsibilities 

USS manager Ensured the USS software was running and undertook troubleshooting when 

needed 

Landing zone safety pilots Served as monitors at beyond visual line of sight landing points 

Flight rest manager Coordinated the crews and flights to conduct the test scenarios properly 

NASA researchers and 

observers 

Collected observational and survey data, observers were available to support 

media day and answer flight team questions 

36 
Appendix B 

UTM Achitecture 

Figure B1. UTM architecture. 

37 
 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix C 

Table C1. Participant and Observer Statement Coding Scheme based on Research Question Topics 

Research Category Sub-category Research Category Sub-category 

Communication/ 

Messaging 

Effectiveness of data (message) 

Safety and Risk 

Safe separation 

Quality of alerting information Confounds 

Usability of 

information -

Information 

efficiency & 

effectiveness 

Reliability and/or accuracy of 

information 

Priority access & priority 

operations 

Appropriateness of decision 

made (maneuver) 

Procedures 

Planning process and 

considerations 

Workload Procedures for flight or event 

Usefulness of information Description of operator role 

Usability tradeoffs Public and hobbyists 

Intuitiveness or interpret-ability 

of information 

Trust 

Confidence in decision made 

(maneuver) 

Easy to understand Trust in information presented 

Information timeliness Reliance on info presented 

Situation 

Awareness 

Transparency of tool Operator buy-in 

Situation Awareness 

Automation 

Definition of terms 

Information required or desired Design of system or hardware 

Saliency of information Functions 

Information timeliness Concept issues 

38 
 

 

 

Appendix D 

General Description of Data 

Observers counted the uses of four types of UTM information from both UTM and other sources. 

There were 190 counts across the three scenarios—61 during scenario 1; 94 during scenario 2; and 

35 during scenario 4. The majority of the observations/counts were for information about the crews’ 

operation (60%). Of the remaining 40%, half of the observations were about UVRs, a quarter about 

negotiation and a quarter about priority operations. 

all scenarios - distribution of count data 

115 19 

18 

38 

own op negotiation priority op UVR 

Figure D1. Observer counts of uses of UTM information by flight crews, n = 190. 

39 

 

 

 

 

 

 

Priority Status 

9 

scenario 2 scenario 4 

Very 

8 

7 

6 

Very 

easy 

Mean rating 

5 

4 

3 

2 

Not 

at all 1

Very 

difficult 

0 

Clarity of change to contingency Accuracy of UTM information Ease of indicating intentions 

-1 landing plan 

Communications about plan changes 

Figure E3. Participant reported properties of messages about plan changes,N-clarity = 

13, n-accuracy = 10, n-indicating intentions = 9, 1-7 scale with scale 

extended to show full standard deviations. 

Conflict Alerting 

None 

Alert seen 

Conflict with UA 

 Conflict with a moving obstacle 

scenario 4 scenario 2 scenario 1 

Conflict with a static obstacle 

0 1 2 3 4 5 

Count 

Figure E4. Alert messages participants recall seeing on UTM, n =14. 

41 

 

 

 

 

 

Appendix F 

Breakout of Negotiation and of UVR Operations 

Table F1. Breakout of Negotiation Operations 

Location 

# of negotiations 

Nevada 

862 

Scenario 

# of negotiations 

Scenario 1 

119 

Scenario 2 

657 

Scenario 4 

86 

Type 

# of negotiations 

Replan 

27 

Intersection 

92 

Replan 

2 

Intersection 

655 

Replan 

4 

Intersection 

82 

Table F2. Breakout of UVR Operations 

Location 

# of UVRs 

Nevada 

24 

Type 

# of UVRs 

Dynamic restriction 

24 

Static advisory 

0 

Scenario 

# of UVRs 

Scenario 1 

4 

Scenario 2 

20 

Scenario 1 

0 

Scenario 2 

0 

43