NASA/TM—2019– 220347 

TCL3 UTM (UAS Traffic Management) Flight Tests,

Airspace Operations Laboratory (AOL) Report 

Lynne Martin 

NASA Ames Research Center 

Cynthia Wolter 

San Jose State University Foundation 

Kimberly Jobe 

San Jose State University Foundation 

Jeffrey Homola 

NASA Ames Research Center 

Michele Cencetti 

Universities Space Research Association 

Quang Dao 

NASA Ames Research Center 

Joey Mercer 

NASA Ames Research Center 

December 2019 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

NASA STI Program...in Profile 

Since its founding, NASA has been dedicated 

to the advancement of aeronautics and space

science. The NASA scientific and technical 

information (STI) program plays a key part in

helping NASA maintain this important role. 

The NASA STI program operates under the 

auspices of the Agency Chief Information Officer.

It collects, organizes, provides for archiving, and

disseminates NASA’s STI. The NASA STI program

provides access to the NTRS Registered and its

public interface, the NASA Technical Reports

Server, thus providing one of the largest collections

of aeronautical and space science STI in the world.

Results are published in both non-NASA channels

and by NASA in the NASA STI Report Series,

which includes the following report types: 

• TECHNICAL PUBLICATION. Reports of

completed research or a major significant

phase of research that present the results of

NASA programs and include extensive data

or theoretical analysis. Includes compilations

of significant scientific and technical data and

information deemed to be of continuing

reference value. NASA counterpart of peerreviewed formal professional papers but has 

less stringent limitations on manuscript length

and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or 

of specialized interest, e.g., quick release 

reports, working papers, and bibliographies 

that contain minimal annotation. Does not 

contain extensive analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored

contractors and grantees. 

• CONFERENCE PUBLICATION. 

Collected papers from scientific and

technical conferences, symposia, seminars, 

or other meetings sponsored or cosponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from

NASA programs, projects, and missions,

often concerned with subjects having 

substantial public interest. 

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific

and technical material pertinent to NASA’s

mission. 

Specialized services also include creating 

custom thesauri, building customized databases,

and organizing and publishing research results. 

For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home page 

at http://www.sti.nasa.gov 

• E-mail your question via to

help@sti.nasa.gov 

• Phone the NASA STI Help Desk at 

(757) 864-9658 

• Write to: 

NASA STI Information Desk 

Mail Stop 148

NASA Langley Research Center

Hampton, VA 23681-2199 

ii 
 

 

 

 

 

 

 

 

 

NASA/TM—2019–220347 

TCL3 UTM (UAS Traffic Management) Flight Tests,

Airspace Operations Laboratory (AOL) Report 

Lynne Martin 

NASA Ames Research Center 

Cynthia Wolter 

San Jose State University Foundation 

Kimberly Jobe 

San Jose State University Foundation 

Jeffrey Homola 

NASA Ames Research Center 

Michele Cencetti 

Universities Space Research Association 

Quang Dao 

NASA Ames Research Center 

Joey Mercer 

NASA Ames Research Center 

National Aeronautics and 

Space Administration 

Ames Research Center 

Moffett Field, California 

December 2019 

iii 
Trade name and trademarks are used in this report for 

identification only. Their usage does not constitute an 

official endorsement, either expressed or implied, by the 

National Aeronautics and Space Administration. 

Available from: 

NASA STI Program

STI Support Services

Mail Stop 148

NASA Langley Research Center

Hampton, VA 23681-2199 

This report is also available in electronic form at http://www.sti.nasa.gov 

or http://ntrs.nasa.gov/ 

iv 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table of Contents 

List of Figures................................................................................................................................. ix 

List of Tables .................................................................................................................................. x 

Acronyms and Definitions ............................................................................................................. xi 

Executive Summary........................................................................................................................ 1 

1. Background................................................................................................................................ 3 

2. Method ....................................................................................................................................... 6 

2.1 Participant Roles and Responsibilities............................................................................... 6 

2.2 Vehicle Characteristics ...................................................................................................... 7 

2.3 Interfaces and Information Displays.................................................................................. 8 

2.4 Method ............................................................................................................................... 9 

2.5 Research Objectives........................................................................................................... 10 

2.6 Test Scenarios.................................................................................................................... 11 

3. Results........................................................................................................................................ 13 

3.1 Metadata............................................................................................................................. 13 

4. Data and Information Exchange Tests....................................................................................... 14 

4.1 Method ............................................................................................................................... 15 

4.1.1 Surveys...................................................................................................................... 15 

4.1.2 Debriefs..................................................................................................................... 15 

4.2 DAT Metadata ................................................................................................................... 15 

4.3 General DAT Feedback ..................................................................................................... 16 

4.3.1 Operator Experience ................................................................................................. 16 

4.3.2 Information Properties .............................................................................................. 17 

4.4 DAT 1: End-to-end UREP................................................................................................. 17 

4.5 DAT 2: FIMS Failover....................................................................................................... 19 

4.6 DAT 3: USS Failover......................................................................................................... 19 

4.6.1 Operator Experiences................................................................................................ 19 

4.6.2 Operator Experience: Situation Awareness .............................................................. 20 

4.6.3 Methods: Procedures................................................................................................. 21 

4.6.4 Information Exchange: Alerting ............................................................................... 21 

4.6.5 Automation ............................................................................................................... 21 

4.7 DAT 4: UAS Identification................................................................................................ 22 

4.7.1 Information Exchange............................................................................................... 22 

4.7.2 Operator: Communications....................................................................................... 23 

4.7.3 Concept ..................................................................................................................... 23 

4.7.4 Automation: Technical.............................................................................................. 23 

v 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

4.8 DAT 5: USS-to-USS Negotiation...................................................................................... 24 

4.8.1 Information Exchange............................................................................................... 24 

4.8.2 Information meets UTM ........................................................................................... 25 

4.8.3 Operator Experiences................................................................................................ 25 

4.8.4 Method ...................................................................................................................... 26 

4.8.5 Automation ............................................................................................................... 26 

4.8.6 Concept ..................................................................................................................... 26 

4.9 DAT 6: Weather Service.................................................................................................... 27 

4.10 DAT Tests Summary ....................................................................................................... 27 

5. Concepts Tests ........................................................................................................................... 28 

5.1 Method ............................................................................................................................... 29 

5.1.1 Surveys...................................................................................................................... 29 

5.1.2 Debriefs..................................................................................................................... 29 

5.2 CON Metadata ................................................................................................................... 29 

5.3 General CON Feedback ..................................................................................................... 30 

5.3.1 Methods: Operational Effectiveness......................................................................... 30 

5.3.2 Information Properties .............................................................................................. 31 

5.4 CON 1: Take-off and Landing while BVLOS................................................................... 32 

5.4.1 Automation: Technical.............................................................................................. 33 

5.4.2 Operator Experience: Situation Awareness .............................................................. 33 

5.4.3 UTM Feedback ......................................................................................................... 34 

5.5 CON 2: Contingency Initiation.......................................................................................... 35 

5.5.1 Information Desired.................................................................................................. 35 

5.5.2 Operator Experience: Situation Awareness .............................................................. 36 

5.5.3 Operator Experience: Decision Making.................................................................... 37 

5.5.4 Operator Experience: Actions................................................................................... 37 

5.5.5 Procedures: Emergency ............................................................................................ 38 

5.5.6 Procedures: Gaming the System............................................................................... 38 

5.6 CON 4: Multiple operations for a sustained period........................................................... 38 

5.6.1 Operator Experience: Situation Awareness .............................................................. 39 

5.6.2 Operator Experience: Workload ............................................................................... 40 

5.6.3 Operator Experience: Roles...................................................................................... 40 

5.6.4 Concept: Procedures ................................................................................................. 40 

5.6.5 Automation: Messaging and Alerting....................................................................... 41 

5.6.6 Automation: Rogue................................................................................................... 42 

5.6.7 Automation: Safety ................................................................................................... 42 

5.6.8 Automation: Using USS and UTM Information....................................................... 43 

5.6.9 Equipment: Telemetry .............................................................................................. 44 

vi 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5.7 CON 5: FIMS/USS Interaction when UAS Heads Towards Unauthorized Airspace ....... 44 

5.7.1 Operator: Situation Awareness ................................................................................. 45 

5.7.2 Operator Experience: Workload ............................................................................... 45 

5.7.3 Information ............................................................................................................... 45 

5.7.4 Information: Alerting................................................................................................ 45 

5.7.5 Warnings................................................................................................................... 47 

5.7.6 Usability of Warnings............................................................................................... 47 

5.7.7 Temporary Flight Restrictions.................................................................................. 47 

5.7.8 Procedures: TFR ....................................................................................................... 48 

5.7.9 Emergency Situations: TFR...................................................................................... 48 

5.7.10 Automation: USS Client Information ..................................................................... 49 

5.7.11 UTM Rules.............................................................................................................. 49 

5.7.12 Automation ............................................................................................................. 49 

5.8 CON Test Summary........................................................................................................... 50 

6. Sense and Avoid Tests............................................................................................................... 51 

6.1 Method ............................................................................................................................... 52 

6.1.1 Surveys...................................................................................................................... 52 

6.1.2 Debriefs..................................................................................................................... 52 

6.2 SAA Metadata.................................................................................................................... 53 

6.3 General SAA feedback ...................................................................................................... 54 

6.3.1 Information Properties .............................................................................................. 54 

6.3.2 Methods: Operational Effectiveness......................................................................... 55 

6.4 Crew Feedback from Debrief Discussions ........................................................................ 56 

6.4.1 Concepts.................................................................................................................... 56 

6.4.2 Technology ............................................................................................................... 58 

6.4.3 Operator Experience ................................................................................................. 60 

6.4.4 Automation ............................................................................................................... 62 

6.4.5 Alerting ..................................................................................................................... 64 

6.5 SAA 1 and SAA 6.............................................................................................................. 65 

6.5.1 Operator Experience: Situation Awareness .............................................................. 65 

6.6 SAA 5 ................................................................................................................................ 66 

6.7 SAA Test Summary ........................................................................................................... 66 

7. Communication, Navigation and Surveillance Tests ............................................................... 67 

7.1 Method ............................................................................................................................... 68 

7.1.1 Surveys...................................................................................................................... 68 

7.1.2 Debriefs..................................................................................................................... 68 

7.2 CNS Metadata.................................................................................................................... 68 

7.3 General CNS Feedback...................................................................................................... 69 

7.3.1 Methods: Operational Effectiveness......................................................................... 69 

7.3.2 Information Properties .............................................................................................. 69 

7.3.3 Technology: RF Spectrum........................................................................................ 71 

vii 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

7.4 CNS 1: Effectiveness of Redundant C2 in Maintaining Operational Control of UA........ 71 

7.4.1 Methods: Management of Failure............................................................................. 72 

7.4.2 Operator Experience: Situation Awareness and Workload ...................................... 73 

7.4.3 Information: Alerting................................................................................................ 74 

7.4.4 Information: SDSP.................................................................................................... 75 

7.4.5 Automation: GCS ..................................................................................................... 75 

7.4.6 Automation: USS Client ........................................................................................... 76 

7.4.7 Automation: USS Issues ........................................................................................... 76 

7.4.8 Concept: Privacy....................................................................................................... 76 

7.4.9 Concept: Participants ................................................................................................ 77 

7.4.10 UTM System........................................................................................................... 77 

7.5 CNS 2: Remaining within Flight Geography using GNSS Navigation............................. 77 

7.5.1 Operator Experiences: Situation Awareness............................................................. 78 

7.5.2 Automation: Technology .......................................................................................... 79 

7.5.3 Automation: USS Client ........................................................................................... 79 

7.5.4 Concept: Safety......................................................................................................... 79 

7.6 CNS 3: RF Interference Baseline Monitoring ................................................................... 79 

7.6.1 Technology: RF Interference .................................................................................... 80 

7.6.2 Automation: Issues.................................................................................................... 80 

7.7 CNS Test Summary ........................................................................................................... 81 

8. TCL 3 Conclusions and Recommendations............................................................................... 82 

9. References.................................................................................................................................. 85 

Appendix 1. Days of Flying for TCL3........................................................................................... 86 

Appendix 2. Crew Size and Composition...................................................................................... 87 

Appendix 3. Vehicles Flown in Flight Test................................................................................... 88 

Appendix 4. List of Flight Tests .................................................................................................... 89 

Appendix 5. Debriefs Conducted................................................................................................... 90 

Appendix 6. Flights Made by Test Sites to Fulfill each Test......................................................... 91 

Appendix 7. Examples of Topics Discussed in DAT Interviews with Flight Crews .................... 92 

Appendix 8. Examples of Notes from DAT Observations ............................................................ 95 

Appendix 9. Concerning Nearby Operations................................................................................. 97 

Appendix 10. Examples of Topics Discussed in CON Interviews with Flight Crews .................. 98 

Appendix 11. Examples of Notes from CON Observations.......................................................... 100 

Appendix 12. Examples of Topics Discussed in SAA Interviews with Flight Crews................... 102 

Appendix 13. Examples of Notes from SAA Observations .......................................................... 104 

Appendix 14. CNS Capability on UAVs....................................................................................... 106 

Appendix 15. Examples of Topics Discussed in CNS Interviews with Flight Crews................... 107 

Appendix 16. Examples of Notes from CNS Observations........................................................... 109 

viii 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

List of Figures 

Figure 1a. Example of a test site with one GCS area for two vehicles........................................... 5 

Figure 1b. Example of a test site with multiple GCS area for its vehicles ..................................... 5 

Figure 2. Illustration of two different types of team organization: Centralized and Decentralized 7 

Figure 3. Suite of tools in use by the AOL research team for TCL3.............................................. 8 

Figure 4. Example of a simple scenario (square survey pattern).................................................... 12 

Figure 5. Example of a more complex scenario with 5 vehicles and more than one USS ............. 13 

Figure 6. Operator experiences during DAT testing in TCL3 shown by type of DAT test............ 16 

Figure 7. UTM information properties during DAT testing in TCL3 shown by type of DAT test 17 

Figure 8. Quality of aspects of UREPs sent as part of DAT 1........................................................ 18 

Figure 9. Operator effort, concerns and recovery during USS failure............................................ 20 

Figure 10. UVIN/ UAS ID data exchange usefulness and timelines.............................................. 23 

Figure 11. Opinions of aspects of the USS-to-USS negotiation process........................................ 25 

Figure 12. User experiences during the USS-to-USS negotiation process..................................... 26 

Figure 13. Operational efficiency for CON flights in TCL3 shown by type of CON test.............. 30 

Figure 14. UTM information properties during CON testing in TCL3 shown by type of CON test 31 

Figure 15. Operator workload during CON testing in TCL3 shown by type of CON test............. 32 

Figure 16. Operator awareness during CON testing in TCL3 for Test Sites 1 and 3 ..................... 33 

Figure 17. Operator awareness during CON 2 testing in TCL3 for Test Site 5 ............................. 37 

Figure 18. Operator awareness and awareness actions during CON 4 testing in TCL3................. 39 

Figure 19. Operator workload during different phases and tasks for CON 5................................. 45 

Figure 20. Types of alert and sources of this information .............................................................. 46 

Figure 21. Information used to stay within flight boundary on SAA flights in TCL3 shown by 

type of SAA test............................................................................................................ 54 

Figure 22. UTM information properties during SAA testing in TCL3 shown by type of SAA test 55 

Figure 23. Operational effectiveness for SAA flights in TCL3 shown by type of SAA test.......... 55 

Figure 24. Awareness of conflict situations during SAA 1 flight tests .......................................... 66 

Figure 25. Operational effectiveness for CNS flights in TCL3 shown by type of CNS test.......... 69 

Figure 26. UTM information properties during CNS testing in TCL3 shown by type of CNS test 70 

Figure 27. Operator workload and UTM awareness during CNS testing in TCL3 shown by type 

of CNS test.................................................................................................................... 71 

Figure 28. Crew management of C2 link failure during CNS 1 test shown by site........................ 73 

Figure 29. Operator monitoring and cross checking (situation awareness) during CNS 1 testing 

in TCL3 and operator workload shown by test site ...................................................... 74 

Figure 30. Operator monitoring and cross checking (situation awareness) during CNS 2 testing 

in TCL3 shown by test site ........................................................................................... 78 

ix 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

List of Tables 

Table 1. Flight Days and Shakedown Days for the TCL3 Flight Demonstration ......................... 4 

Table 2. Crew Member Roles and Responsibilities ...................................................................... 6 

Table 3. Test Site Support Personnel Roles and Responsibilities................................................. 6 

Table 4. Information Topics used for Organizing Qualitative Data Collected ............................. 10 

Table 5. Test Objectives that Each Test Site was Awarded with the Date(s) on which they Flew 

Missions to Meet those Objectives ................................................................................. 11 

Table 6. Number of Live and Simulated Flights Providing Data for TCL3 by Test Site ............ 14 

Table 7. Number of Live and Simulated Flights Providing Data for TCL3 by DAT test............. 15 

Table 8. DAT 1: Discussion Topics and their Notional Categorization into Information Themes 17 

Table 9. DAT 3: Discussion Topics and their Notional Categorization into Information Themes 19 

Table 10. DAT 4: Discussion Topics and their Notional Categorization into Information Themes 22 

Table 11. DAT 5: Discussion Topics and their Notional Categorization into Information Themes 24 

Table 12. Number of Live and Simulated Flights Providing Data for TCL3 by CON Test ........ 30 

Table 13. CON 1: Discussion Topics and their Notional Categorization into Information Themes 32 

Table 14. CON 2: Discussion Topics and their Notional Categorization into Information Themes 36 

Table 15. CON 4: Discussion Topics and their Notional Categorization into Information Themes 39 

Table 16. CON 5: Discussion Topics and their Notional Categorization into Information Themes 44 

Table 17. SAA Discussion Topics and their Notional Categorization into Information Themes. 53 

Table 18. Number of Live and Simulated Flights Providing Data for TCL3 by SAA Test.......... 53 

Table 19. Reasons for Not Relying on UTM for Safety ............................................................... 56 

Table 20. Number of Live and Simulated Flights Providing Data for TCL3 by CNS Test.......... 68 

Table 21. CNS 1: Discussion Topics and their Notional Categorization into Information Themes 72 

Table 22. CNS 2: Discussion Topics and their Notional Categorization into Information Themes 78 

Table 23. CNS 3: Discussion Topics and their Notional Categorization into Information Themes 80 

x 
 

 

 

 

 

 

 

 

 

 

Acronyms and Definitions 

2D...........................................Two Dimensional 

4D...........................................Four Dimensional 

ADS-B ...................................Automatic Dependent Surveillance-Broadcast 

AGL .......................................Above Ground Level 

AOL .......................................Airspace Operations Laboratory 

BVLOS ..................................Beyond Visual Line-Of-Sight 

C2...........................................Command and Control 

CNS........................................Communications, Navigation and Surveillance 

CON .......................................Concept set of tests 

COTS .....................................Commercial Off the Shelf Software 

DAA.......................................Detect and Avoid 

DAT .......................................Data and Information Exchange set of tests 

DMP.......................................Data Management Plan 

DoT ........................................Department of Transport 

DSRC .....................................Dedicated Short Range Communications 

FAA .......................................Federal Aviation Administration 

FIMS ......................................Flight Information Management System 

ft .............................................Feet 

GA..........................................General Aviation 

GCS........................................Ground Control Station 

GCSO.....................................Ground Control Station Operator 

GDOP ....................................Geometric Dilution of Precision (of GPS signal) 

GHz ........................................Gigahertz 

GNSS .....................................Global Navigation Satellite System 

GPS ........................................Global Positioning System 

GUFI ......................................Globally Unique Flight Identifier 

GUI ........................................Graphical User Interface 

HDOP ....................................Horizontal Dilution of Precision (of GPS signal) 

ID ...........................................Identification 

iOS .........................................Apple company operating system 

ISM ........................................Industry, Scientific and Medical 

ISN .........................................Internet Services Network 

iUTM......................................insight UTM Visualization tool 

LOS........................................Line-of-Sight 

LTE ........................................Long Term Evolution 

LUN .......................................Local USS Network 

LZ...........................................Landing Zone 

m/s..........................................Meters per Second 

MSL .......................................Mean Sea Level 

xi 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

NASA.....................................National Aviation and Space Administration 

nc............................................National Campaign 

nmi .........................................Nautical Mile 

NOAA....................................National Oceanic and Atmospheric Administration 

NUSS .....................................NASA UAS Service Supplier 

PIC .........................................Pilot-in-Command 

PII...........................................Personally Identifiable Information 

PKI.........................................Public Key Infrastructure 

RC ..........................................Radio Controlled 

RF...........................................Radio Frequency 

RTB........................................Return to Base 

RTT........................................Research Transition Team 

SA ..........................................Situation Awareness 

SAA .......................................Sense and Avoid 

SD ..........................................Situation Display 

SDSP......................................Supplemental Data Service Provider 

sec ..........................................Second(s) 

SOW.......................................Statement of Work 

TCL........................................Technology Capability Level 

TCL2-nc.................................Technology Capability Level 2 National Campaign 

TFR ........................................Temporary Flight Restriction 

UA..........................................Unmanned Aerial Vehicle 

UAS .......................................Unmanned Aircraft System 

UAV ......................................Unmanned Aerial Vehicle 

UFO .......................................Unidentified Flying Object 

UREP .....................................UAS Pilot Report 

USS ........................................UAS Service Supplier 

USS Op ..................................USS System Operator 

UTM.......................................UAS Traffic Management 

UUID......................................Universal Unique Identifier 

UVIN......................................UAV Vehicle Identification Number 

VLOS.....................................Visual Line-of-Sight 

VO..........................................Visual Observer 

Wi-Fi......................................IEEE 802.11x 

Wx..........................................Weather 

xii 
 

 

 

 

 

 

 

TCL3 UTM Flight Tests, 

Airspace Operations Laboratory (AOL) Report 

Lynne Martin1, Cynthia Wolter2, Kimberly Jobe2, Jeffrey Homola1, 

Michele Cencetti3, Quang Dao1, Joey Mercer1 

Executive Summary 

The Technology Capability Level-3 (TCL3) flight tests were conducted at six different test sites 

located across the USA from March to May of 2018. The campaign resulted in over 830 data 

collection flights using 28 different aircraft and involving 20 flight crews. Flights not only varied in 

duration, but also in the environments and terrains over which they flew. The TCL3 tests highlighted 

four different types of tests: three tests focused on Communication, Navigation and Surveillance 

(CNS); six tests focused on Sense and Avoid (SAA) technologies; six tests focused on USS Data and 

Information Exchange (DAT); and five tests focused on exploring fundamental Concepts of the 

project (CON). This document presents data collected during the TCL3 tests that informed the 

operator’s experiences—the quality of the unmanned aerial system (UAS) Service Supplier (USS) 

information that the operator was provided with, the usefulness of this information, and the usability 

of the automation, both while airborne and on the ground. It is intended to complement the reports 

written by the test sites and the quantitative reports and presentations of the UAS Traffic 

Management (UTM) project. 

With the goal of instructing what the minimum information requirements and/or best practices might 

be in TCL3 operations, the driving enquiry was: How do you get the information you need, when 

you need it, to successfully fly a UAS in UTM airspace? This enquiry touches on two requirements 

for displays, which are to provide adequate situation awareness (SA) and to share information 

through a USS. 

The six test sites participating in the TCL3 tests flew a subset of the 20 tests (outlined above), with 

most sites working on a subset of each of the four types: Communications, Navigation and 

Surveillance (CNS); DAT; CON; and Sense and Avoid (SAA). The, mainly qualitative, data 

addressed in this report was collected by the AOL (Airspace Operations Laboratory) both on-site and 

remotely for each test. The data consists of the contents of end-of-day debriefs, end-of-day surveys, 

observer notes, and flight test information, all submitted as part of the Data Management Plan 

(DMP). 

The topics of inquiry were selected via a process of analyzing the human-automation interaction 

1 NASA Ames Research Center; Moffett Field, California. 

2 San Jose State University Foundation, Moffett Field, California. 

3 Universities Space Research Association 

1 
 

 

 

 

 

 

 

 

 

 

 

elements of each TCL3 test, which were gleaned from the statement of work and test site proposals. 

Elements of interest were grouped under six major categories to facilitate later data analysis. These 

six categories were: the information that UTM users to complete their tasks; the characteristics of the 

UTM/ground station displays that facilitated the use of information, as this is where information 

meets the user; the state or experience of the operator while working in the UTM environment— 

their level of load and awareness, in particular; the functionality and usefulness of the UTM-related 

automation in the flight tests; the methods the operators used to make use of that automation and 

UTM information; and, the exploration of the UTM concept during each flight test. 

There were up to three teams of researchers collecting data from the participants during field test 

days. For at least one of each sites’ test weeks, an AOL researcher was physically present at each test 

site to collect data. To the extent possible, researchers observed all flight crews at some point across 

the test days. In the AOL, a team of UTM developers verified data flowing through the system and 

teams of three or four AOL researchers remotely collected data from the participants at each test site 

about their experiences. 

During the 831 test flights of the TCL3 effort, participants provided a number of positive points of 

feedback within 280 individual survey responses and 40 debriefs conducted with crews at test sites. 

Data from CNS and DAT tests were very helpful, and the tests were completed as expected. The 

tools investigated as part of the SAA tests were well-received by operators and helped with their 

overall situation awareness. CON tests were interesting and varied, and generated a good deal of 

data while flying different vignettes and afterwards at broad concept-level discussions. From this 

wealth of data, 11 points are highlighted about the information required, information usage, operator 

approach, automation, concept, and UTM methods or procedures that operators had worked with 

during the flight tests. 

• Test sites appreciated the additional sensor tools that provided increased situation 

awareness about the airspace they were flying in. Telemetry that provides such situation 

awareness to operators came from a range of sensors, and one type of data was not 

distinctly better than another when comparing across tests. However, it was emphasized 

that the data provided needs to be reliable and consistent. 

• Crews stressed that command and control (C2) link displays should be both simplified and, 

if possible, situated on the USS client. Moving forward, indication of the strength of the C2 

connection might be added to USS displays. 

• The interfaces for generating outgoing and reading incoming messages including those for 

alerting need to be improved. Clutter needs to be reduced in all displays. 

• Users were concerned about being flooded with messages if every non-normal UAS reading 

were to be broadcast to the community. This led to a wider discussion of how much is “too 

much” for all types of alerting. Crews differed in their opinions of UTM alerting depending 

on the situation. They wanted to be alerted as soon as possible to events that could affect 

their flight or airspace, but did not want to be distracted by repeated alerts or warnings that 

were “obvious” and suggested that the community specify some lower limits on this. 

• Although users admitted to not wanting to share their data/ information with other 

companies/operator groups, they agreed that some level of sharing has to happen to make 

UTM useful. Implications are that the community should specify lower limits on what 

needs to be broadcast, outlining information that must be shared and when. 

• Crews understood that maintaining an awareness of DAT messages and USS interactions 

was both useful and important. 

2 
 

 

 

 

• Although participants appreciated the ability of their SAA tools to alert them regarding the 

nearness of approaching vehicles, they reported a number of issues with this new-to-UTM 

technology, ranging from telemetry reliability to interface interaction issues. They 

concluded, in general, that these tools need more development to become truly useful. 

• Procedures for reservations and negotiations could be more formalized. UTM users were 

able to “step on” each other to reserve airspace, and some did this accidentally, while 

others gamed the system in order to gain an advantage. These loopholes should be 

identified and solutions considered, where incentives could promote efficiency, and 

feedback about the status of applications needs to be improved. 

• The proper method to respond to SAA events was an area of uncertainty for crews. 

Discussion highlighted the variety of approaches crews considered that might be used to 

respond to conflict situations. 

• Although participants did not question the need for Temporary Flight Restrictions (TFRs), 

they indicated that the rules and procedures governing them were unclear. Rules for when 

TFRs will be generated need to be clarified, as do exit times and procedures for exiting, 

and who, if anyone, has clearance to remain within a TFR. 

• Rules or procedures for sending messages and methods for interacting with messages need 

to be formalized. 

These themes each relate directly to one of the six overarching categories described above that were 

the focus topics for this effort, and guided the data collection process, to inform the AOL’s question: 

“What information is required to successfully fly in a UTM environment?” As outlined in the points 

above, under the first theme of information properties, alerting was a key topic of discussion and 

concern to many of the operators; where information meets the user, it was found that the 

transparency of the UTM and USS tools was extremely important. Generally, operators indicated 

that they liked the tools that presented data clearly and reliably in this study, but felt that further 

training would be needed to increase SA, acceptance of the UTM concept, and operator willingness 

to participate in such an environment. Operators’ situation awareness was both influenced by the 

usability of the information presented by a crew’s window into UTM—their USS client—but also by 

how aware they were of what information was accessible to them. Further development and research 

on possible procedures, standards, and/or recommendations are needed. All of these themes together 

help us understand to how UTM was used, what went well and should persist, and how the UTM 

experience can be improved. In sum, the TCL3 flight tests were successful, and more was learned 

about the operators, the information they need to fly within UTM, and the procedural requirements 

in a TCL3 environment. 

1. Background 

As part of NASA’s Unmanned Aircraft Systems (UAS) Traffic Management (UTM) effort 

(Kopardekar, et al., 2016), the Technology Capability Level-3 (TCL3) flight demonstrations took 

place across three months from March through May 2018 and involved six partner groups, or test 

sites, located across the USA. Those three months encompassed 37 calendar days on which vehicles 

flew test flights, sometimes at more than one test-site concurrently (see Table 1 and Appendix 1). 

There were 10 official shakedown (i.e., ‘practice’) flying days and 51 flying days for data3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

collection2. Each test-site was utilized and configured to meet the needs of the vehicles and the 

criteria specified in the test scenarios written by test sites to fulfill the criteria set out in the NASA 

Statement of Work (SOW, Rios, 2017). Some test sites had as many as five ground control station 

(GCS) locations, from which flight crews conducted their operations, while others had one, 

depending on the test type (see Figure 1a and Figure 1b for examples). Some test sites moved the 

locations of their GCSs depending on the tests they were flying that day, while others were fixed 

bases. Flight crews varied in composition and size but generally were smaller than for the 

Technology Capability Level-2 national campaign (TCL2-nc) (Martin, et al., 2018). Flight crews 

from some test sites were composed of individuals from one organization, while other test sites sent 

multiple crews, each from different teams. Depending on the test, half to two-thirds of the test sites 

(sites 1, 4, 5 and sometimes 6) centrally managed their UTM service supplier (USS) onsite, with one 

USS system operator (USS Op), in some cases located separately from the flight crews, overseeing 

the USS operations for a number (or all) of that test-site’s crews. The other test sites integrated a 

USS Op within each flight crew. Scenarios were developed by each test-site to demonstrate the 

UTM capabilities that they had proposed. Most test sites created multiple unique scenarios to meet 

the variety of test requirements. 

Table 1. Flight Days and Shakedown Days for the TCL3 Flight Demonstration 

Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 

Week 1 

Week 2 

Week 3 

Week 4 

Week 5 

Week 6 

Week 7 

Week 8 

Week 9 

Week 10 

Week 11 simulation only 

Week 12 

Week 13 

Shakedown days 1 1 4 0 3 1 

Test days 11 4 6 9 13 7 

Total days in field 12 5 10 9 16 7 

Key: orange fill = a shakedown day; green fill = a data-collection day; blue fill = a simulation only day. 

2 One site retested some of their data collection in the lab—hence there were 51 days of testing for data collection with 

50 of these flying vehicles in the field and one simulating all flights in the lab. 

4 
 

Figure 1a. Example of a test site with one GCS area for two vehicles (colored lines 

denote vehicle volume plan-areas). 

Figure 1b. Example of a test site with multiple GCS area for its vehicles (colored 

lines denote vehicle volume plan-areas). 

5 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2. Method 

2.1 Participant Roles and Responsibilities 

Flight crews varied in number and affiliation: some consisted of just one individual, while others had 

up to six in their crews (Appendix 2). Primary flight crew positions are listed in Table 2, with many 

crew members fulfilling more than one role. Additional positions staffed by some, if not all, of the 

flight test sites are listed in Table 3. 

Table 2. Crew Member Roles and Responsibilities 

Crew Member Role Crew Member Responsibilities 

Pilot-in-command Serve as the main pilot for the vehicle 

GCS operator Work the vehicle’s flight planning and flight 

execution software 

USS operator Monitor and interact with USS displays (NASA) 

Hardware and software flight 

engineers Support specific technical aspects of the vehicle 

Visual observers Safety monitors who provide visual contact with 

the vehicles at all times 

Table 3. Test Site Support Personnel Roles and Responsibilities 

Test Site Personnel Role Responsibilities of this Role 

USS manager 

Ensure the USS software was running and 

undertook troubleshooting when needed 

Radio control safety pilots Serve as alternate pilots if the PIC needed 

assistance 

Flight Test Manager Coordinate the crews and flights to conduct the 

test scenarios properly 

NASA researchers/observers 

Collect observational and survey data, observers 

were available to support media day and answer 

flight team questions 

Although each test-site created their own configurations of personnel, two types of team 

organization persisted from the previous flight test, TCL2-nc, with respect to UTM (Figure 2). 

One type of team organization included having the USS Op role as a dedicated member of the flight 

crew, either completing USS client management tasks alone, or by having one crew member 

splitting the USS Op role with another role (e.g., at Test-Site 3 the flight crew consisted of two 

people: a GCSO/PIC/USS Op and a safety pilot/launch engineer). The advantages of having the USS 

Op role within a flight crew team was that this person was able to focus completely on the crew’s 

mission and communications were reduced. The cost was the number of additional personnel, or, if 

the role was timeshared by one team member, that periods of high workload were compounded if all 

roles were busy at the same time, (e.g., at launch). The second type of team organization was one in 

which a dedicated USS Op fulfilled that role for a number of crews (e.g., at Test-Site 1, one USS Op 

6 
 

 

 

 

 

 

 

 

 

 

 

submitted and managed the 4D plan-areas for the flight, also called flight “volumes,” for four flight 

crews, where each flight crew consisted of a PIC, a GCSO, and a launch engineer). Three test sites 

took this “hub-spoke” approach—Test sites 1, 4 and 5. One test site was a hybrid where some crews 

were co-located, and others were not (because multiple USS were being used). The advantage of 

separating out the USS Op role was that this person became a specialist and overall required 

manpower was reduced. The cost was the increase in communications load as the USS Op had to 

stay in contact with all the flight crews they were serving, and the workload related to managing 

multiple flights in the case that one flight crew/vehicle was having an off-nominal event. 

Centralized GCS Decentralized GCS and Mission Control 

USS Op 

PIC 

VO GCSO 

Eng. 

USS Op 

PIC 

GCSO 

VO 

Eng. 

Test 

manager 

USS 

Manager 

GCS 

Mission 

control 

Figure 2. Illustration of two different types of team organization: Centralized 

and Decentralized. 

2.2 Vehicle Characteristics 

The vehicles flown during the demonstration were a mix of small fixed-wing, multi-rotor, and hybrid 

UAS vehicles, each with varying performance characteristics and endurance limits. There were 26 

different models of aircraft flown (Appendix 3). The multi-rotor vehicles were able to take-off and 

land vertically in a small area and turn on a point in the air, while the fixed-wing vehicles flew 

similarly to manned light-aircraft, taking off on a climbing trajectory, making banked turns in the air, 

and either gliding down into a belly landing or descending to a lower altitude before deploying a 

parachute. All of these methods required larger areas on the ground than the multi-rotor vehicles. 

Hybrid vehicles performed similarly in-flight to fixed-wing vehicles, but with the vertical take-offs 

and landings of multi-rotor vehicles. 

Most vehicles could be controlled either by providing point-to-point direction through a GCS or 

manually by a pilot-in-command (PIC), although some were fully-automated only (GCS control 

only). As an example of the latter, Test-Site 3 managed their operations with automated control only 

(a safety pilot was on hand to take over manual control in an emergency). During all flights, whether 

they were line-of-sight (LOS) or beyond visual line-of-sight (BVLOS), the behavior of the vehicle 

was monitored by at least one visual observer (VO) at all times3. In cases of unexpected vehicle 

behavior, the visual observer could contact the flight crew and/or the Flight Test Manager so that the 

appropriate compensatory action could be taken. 

3 Multiple VO were positioned along BVLOS routes to ensure “eyes on” the vehicles at all times. 

7 
 

 

 

 

 

 

 

 

2.3 Interfaces and Information Displays 

Equipment available at each GCS location varied widely across and within test sites. At most GCSs, 

several displays were available to the flight crews to give them information about their vehicle’s 

flight, and some also included displays to show surrounding operations, and/or aspects of the UTM 

system. For example, Test-Site 3 provided four screens for its GCSO/PIC/USS Op. This individual 

did not have LOS contact with (could not see) their vehicle. Standard tools shown on their displays 

were their flight planning/execution software, a USS client, and a fusion of radar, multi-lateration 

systems and GCS telemetry. The fourth screen was available for use to show other information of 

the GCSO/PIC/USS Op’s choice, including weather, vehicle and USS data, radio frequency usage, 

etc. Other test sites, which had more mobile/portable GCSs, used fewer displays. At Test Site 5, for 

example, flight crews only had a hand-held controller, and one display showing the autopilot 

software for their vehicle. These flight crews did not have access to a display of UTM information. 

Instead, UTM information was verbally relayed to them from a nearby centralized location, where 

the USS Op had such a display. 

All test sites used at least one surveillance system to provide information about the airspace not 

provided by vehicles’ on-board sensors (GPS: global positioning system; ADS-B: Automatic 

Dependent Surveillance-Broadcast), helping to identify other manned and unmanned aircraft flying 

near the test-site and most added radar capabilities or other tools that they used to complete test 

requirements. During the testing, a NASA-built iOS application (insight UTM, or iUTM), provided 

visualizations of UTM system information and current operations, and was made available to the test 

sites. Test Site 3 elected to use iUTM as an additional situation awareness display. Another in-house 

situation display (SD) tool facilitated data collection within the AOL at NASA Ames Research 

Center, allowing the research team to check flight details and monitor UTM messages in real time 

(see Figure 3). 

Figure 3. Suite of tools in use by the AOL research team for TCL3. 

8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In the same way that there was a mix of team members and vehicle types, the partner-built interfaces 

to UTM also differed. Across the test sites, five different partner-built UTM Service Suppliers 

(USSs) and the NASA USS were employed during the flight demonstration. Four test sites used 

more than one USS (Test sites 2, 4, 5, and 6), and two test sites used the same USS (Test sites 1 and 

4). The tools and displays available within these USSs varied, primarily because each partner 

developed their USS independently, with no standard regarding how to display various pieces of 

information. The USSs were still under development and had a wide variety of available functions 

and features. To participate in TCL3, all USSs needed to have certain basic capabilities (i.e., 

capabilities to allow proper FIMS/USS and USS/USS communications, plan submissions, position 

reporting, messaging, and alerting), but the manner and extent by which the partners met those 

requirements differed, and are not examined in this paper. 

2.4 Method 

During test days, there were up to three teams of researchers collecting data from the participants in 

the field. In the NASA labs, a team of UTM developers verified data flowing through the system and 

teams of three or four AOL researchers remotely collected data from the participants at each test-site 

about their experiences and during shakedowns. To the extent possible, researchers observed all 

flight crews at some point across the test days. In addition, for at least one of their test weeks, an 

AOL researcher went to the field to collect data. Data were collected in a number of ways: 

• observations of the participants during flights 

• end-of-day surveys 

• end-of-day group debriefs 

All of these methods solicited feedback on the six areas of interest for these tests (Table 4). In total, 

34 end-of-day group debriefs were collected across the six test sites, some in person and some over 

the phone. Test-specific debrief questions were pre-defined and asked along with alternate questions 

that probed into unique events of that particular day. During these end-of-day debriefs, flight crews 

discussed the UTM-specific topics as they related to the UTM operations at their test-site. Survey 

items were generated with the four topics (Table 4) in mind, but were presented to the participants in 

the context of the flight tests that the Test Site had just flown. Approximately 30–54 questions were 

generated across four surveys, but conditions were set so that participants only answered around 25 

at any one time. Most questions used a seven-point rating format, with 7 representing a very positive 

rating and 1 representing a very negative rating, but some questions were multiple choice or openended. Researchers in the field also took notes while they were watching flight tests. Two sources of 

operational data were also obtained: 

• test sites telemetric flight data, and sometimes other logged data, were shared 

with NASA 

• NASA’s internal records of USS data, were captured through a separate, dataaggregation server 

9 
 

 

 

 

 

 

 

Table 4. Information Topics used for Organizing Qualitative Data Collected 

2.5 Research Objectives 

With the goal of instructing what the minimum information requirements and/or best practices 

might be in TCL3 operations, the driving enquiry was: How do you get the information you need, 

when you need it, to successfully fly a UAS in UTM airspace? This enquiry touches on the 

requirement for displays to: 

• provide adequate situation awareness (SA) 

• share information through a USS 

And, the additional requirement for operators to: 

• have enough knowledge in order to understand what they are seeing 

• respond quickly enough when an action is needed 

There are six overarching categories in the consideration of information required to fly under UTM: 

information properties, information as it interacts with the user, the operators’ responses, methods, 

automation, and concept issues (Table 4). Each of these higher-level categories were split into 

between three and seven subcategories. 

• Information Properties: concerned with features of the information items themselves, 

such as saliency, usability and intuitiveness of information items 

• Information meets User: concerned with tool features that made the information more 

or less usable and the operator’s interaction with the tools 

• Operator Status: concerned with the impact of the information on the state or activity of 

the crews, such as workload and situation awareness 

• Methods: concerned with the procedures and processes that crews put in place to 

interact with UTM and achieve their flights 

10 
 

 

 

 

 

 

 

 

 

• Automation: concerned with the way the automation worked and functions that were 

present or desired 

• UTM Concepts: concerned with the wider world of UTM, encompassing how different 

users might interact with the system and issues that require a higher-level policy or 

concept consideration not just an automation solution (see FAA, 2018 or Kopardekar, 

et al., 2016 for a description of the UTM concept) 

The first four categories of the matrix (information properties, information meets user, operator 

status and methods) were included in Revision F of the Data Management Plan (DMP January 2018) 

that was constructed to inform and assist Test Sites with their data collection process. These 

categories were used to guide organizing the comments from debrief discussions and field notes. 

2.6 Test Scenarios 

The test aims were presented to the sites as twenty test objectives: three CNS; six Sense and Avoid 

(SAA) technologies tests; six USS Data and Information Exchange (DAT) tests; and five Concepts 

(CON) issues tests. The six test sites opted to look at different sets of objectives, and different 

numbers of tests were awarded to each. The tests completed by each test site are shown in Table 5. 

Table 5. Test Objectives that Each Test Site was Awarded with the Date(s) on which 

they Flew Missions to Meet those Objectives 

Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 

General 

Shakedown 

CNS 1 25, 27 Apr 6, 7, 9 Mar 10, 11 Apr 11, 12 Apr 5, 6 Apr 

CNS 2 30 May 6, 8 Mar 5, 6 Apr 

CNS 3 7, 9 Mar 24 Apr 9, 10 Apr 5, 6, 16 Apr 

SAA 1 3, 5 Apr 

SAA 2 23 Apr 23 Apr 

SAA 3 10 May 10, 11 Apr 

SAA 4 10 May 7, 9 May 

SAA 5 11 May 10, 11 Apr 

SAA 6 3, 5 Apr 

DAT 1 6–9 Mar 16 Apr 19 Apr 24 Apr 

DAT 2 6–9 Mar 16 Apr 

DAT 3 6–9 Mar 16 Apr 19 Apr 24 Apr 

DAT 4 11 Apr 16 Apr 24 Apr 

DAT 5 6–9 Mar 16 Apr 19 Apr 24 Apr 

DAT 6 6–9 Mar 16 Apr 24 Apr 

CON 1 4, 17 May 17–19 Apr 24 Apr 

CON 2 25 May 15 Apr 25 Apr, 15 May 

CON 4 23, 25 May 19 Apr 26 Apr 25 Apr, 15 May 

CON 5 16, 18 May 22 Mar 27 Apr 

11 
 

 

 

 

 

The 20 tests were designed to give insight into specific aspects of the UTM concept, each focusing 

on a different problem or interest. CNS tests focused on data interference and blocking (see 

Appendix 4 for a listing), SAA tests focused on different methods to sense and avoid other vehicles, 

DAT tests focused on the functioning of the UTM system, looking at the impact of adding 

information to the system and the impact of system degradation. CON tests looked at aspects of the 

way UAS may potentially be used, from long flight times to unintentionally breaching restricted 

airspace. Combined, these tests touched on most of the themes from the National Campaign flight 

tests of 2017 (BVLOS operations, dynamic re-planning, responses to alerts from the UTM System, 

and the implementation of off-nominal contingency plans). 

With the direction given in the SOW (Rios, 2017), test sites created their own test scenarios to 

investigate one or more of the specified tests. Procedures were not firmly defined in the SOW, and 

test sites were allowed to interpret the document in their own way. Site differences were expected 

and encouraged, and test sites built scenarios that were as large or small in scope as required to 

investigate the questions put forward by the SOW. For those addressing CNS issues, test sites were 

able to address their objectives with several, small variations of a single, simple scenario (see 

example in Figure 4). For CON tests, in general, the scenarios had to be more complex, involving 

multiple vehicles interacting within close proximity (see example in Figure 5). The test sites’ local 

geography and environment also influenced their test scenarios. For example, some GCSs were at 

airfields while other locations were in farmers’ fields, and some GCSs had tree cover, while others 

were on marshy ground close to water, which influenced the scenarios that were created and the 

“stories” that sites tried to tell. 

Figure 4. Example of a simple scenario (square survey pattern). 

12 
 

Figure 5. Example of a more complex scenario with 5 vehicles and more than one USS. 

Note: Magenta polygons are active volumes, cyan polygons are accepted volumes and the 

brown area is the USS client coverage area. The position of an airborne UAS is visible in 

the lower left corner/volume. 

3. Results 

3.1 Metadata 

In the TCL3 tests, sites were required to define specific information about each operation along with 

each submitted operation into UTM. Some of the required fields included the test site, the exact 

test(s) being conducted, if the operation was intended for data collection (as opposed to debugging or 

shakedown), and whether the operation was a live flight or if it was simulated. As these data were 

input manually before flight, and some partners had more capability to alter the content than others, 

there is a margin of error in the accuracy of what was originally reported. NASA employed two 

efforts to increase the accuracy of the data. During each flight, a NASA researcher would crosscheck 

the data in the lab and query the test site if there was a concern, then publish an amendment to the 

submitted data. Then, once all tests by all test sites were completed, researchers crosschecked 3,020 

rows of data and 2,201 submitted operations to define the set of 831 operations that are currently 

being considered as valid data collection flights. However, it should be noted that error still remains 

within these data. One of the concerns is that, although operations might have only one test listed, 

that test may have been run simultaneously with another (i.e., DAT 5 during CON 4); therefore, the 

counts of flights per test may be askew. Secondly, 35 operations do not have a test specified in the 

data. A third concern is operations that were intended for data collection when submitted, but were 

later scrubbed and re-flown without notice and without changing the previous marker from “data 

collection”. 

13 
 

 

 

 

 

 

 

 

 

 

 

 

With the above error sources in mind, the current data set for TCL3 operations includes a total of 

831 operations submitted to UTM for data collection (Table 6). Of these, NASA submitted 43 

simulated operations to virtually interact with the test sites during four specific tests (DAT 3, DAT 5, 

CON 2, CON 4, CON 5). The six test sites submitted 788 flight activities, 627 of which were live 

flights and 161 were simulated flights. Data for days that were identified as shakedowns for 

hardware and software testing have been excluded from the final analysis. 

Table 6. Number of Live and Simulated Flights Providing Data for TCL3 by Test Site 

Test Site NASA Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Totals 

Live flights 0 94 49 58 111 48 267 627 

Simulated flights 43 0 59 0 0 0 102 204 

Total submitted 

operations 43 94 108 58 111 48 369 831 

4. Data and Information Exchange Tests 

The DAT tests were created to explore the exchanges between key system components in both 

nominal and off-nominal situations. These were designed to capture system-level robustness and 

usability through detailed evaluation of individual components and their interactions. Specifically, 

the DAT effort looked at the following three interactions: USS-Operator; USS-USS; and Flight 

Information Management System (FIMS)-USS, through six unique tests (DAT 1 through DAT 6). In 

the End-to-End UAS pilot report (UREP) test (DAT 1) operators submitted UREPs while 

completing other tests. Some elements that were evaluated were the usability of the data to the 

human operator and the appropriateness of the data elements. The FIMS Failover test (DAT 2) 

simulated FIMS, or a key FIMS component, failing while there were active flights. Recovery time 

and the effects on operations were measured. The USS Failover test (DAT 3) simulated USS, or a 

key USS component, failing while there were active flights under its management. Recovery time 

and the effects on operations were measured. For the UAS Identification test (DAT 4) UAVs were 

registered with UTM public key infrastructure (PKI) and broadcasted identity information. This 

identification (ID) was received on the ground and others aimed to identify the flight. The general 

effectiveness of this procedure was the focus of this test. In the USS-to-USS Negotiation test (DAT 

5), two operations with overlapping geographical/temporal areas were submitted to UTM, each using 

a different USS to plan and execute a mission. The USS of the second operation had to initiate a 

negotiation with the USS of the first operation to coordinate a strategic resolution of the conflict, 

either by space, by time, or both. Test sites used both automatic and manual methods to detect and 

resolve the conflicts. The Weather Service test (DAT 6) was designed for crews to use a UTMfocused weather service to aid in the planning of an operation. While the test could be completed in 

conjunction with other tests, the weather data needed to be provided via a defined and published 

interface. 

Five of the six test sites were awarded one or more DAT tests, with two sites being awarded all six 

DAT tests. In total, 22 DAT tests were awarded, and these awards were distributed so that each DAT 

test was performed by either three or four test sites (see Table 7). Test sites conducted these 22 tests 

over 10 calendar days, often performing them in parallel with other tests. 

14 
 

 

 

 

 

 

Table 7. Number of Live and Simulated Flights Providing Data for TCL3 by DAT Test 

NASA Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Total 

DAT 1 12 8 6 26 

DAT 2 9 4 1 14 

DAT 3 3 21 6 5 3 38 

DAT 4 14 5 19 

DAT 5 13 38 17 7 53 128 

DAT 6 10 10 

All DAT 16 90 14 40 18 57 235 

Note: Colored cells with no flight number identify where the number of operations to fulfill 

this test are unknown 

4.1 Method 

4.1.1 Surveys 

There were 54 questions in the DAT survey, with the questions preselected to show only between 

15 and 21 of these questions depending on the test, e.g., those completing DAT 1 saw 19 

questions—10 general and 9 specific to the DAT1 test—as well as a couple of information 

questions. Many questions asked the participant to rate their answers on a 1 to7 scale, where 7 was 

high, or positive, and 1 was low, or negative. Other question types were multiple choice and freeresponse. One hundred and eleven surveys were started in total across the five sites. All 111 surveys 

had some data entered but not all were complete. The greatest number of surveys started at one site 

was 47 and the fewest was six, but note that, although three sites undertook five or more DAT tests, 

one site only undertook one. 

4.1.2 Debriefs 

There were eleven debriefs that focused on DAT tests hosted across the five test sites. Six were 

specific debriefs focused on one DAT test while the other five touched on more than one DAT test 

(see Appendix 8). Numbers of discussion prompts varied as researchers wanted crews to discuss and 

explore the topics within the time available. Prompts were on the topics of operator situation 

awareness, the procedures for UREPs and negotiation, what crews had gained from the tests and 

technical issues of concern. 

4.2 DAT Metadata 

There were 235 operations submitted to UTM for DAT testing, 16 of which originated from the 

NASA AOL for collaboration with the test sites, and 219 originated from the test sites themselves. 

DAT 1 had 26 operations submitted by the test sites, DAT 2 had 14, DAT 3 had 35, DAT 4 had 19, 

DAT 5 had 115, and DAT 6 had 10. As shown in Table 7, there are gaps in the data where it is likely 

that the DAT tests were run during the same flight as another test, but this was not noted in the logs. 

15 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mean rating 

4.3.2 Information Properties 

On average, participants rated all the properties of the UTM information they received as “good” 

with means varying from 4.7 for clarity of information to 5.3 for accuracy of information. 

Participants gave the highest ratings (on average) after DAT 5 (USS-USS negotiation) and the 

lowest after DAT 4 (UVIN/UAS ID exchange) (Figure 7). 

8 DAT1 DAT2 DAT3 DAT4 DAT5 

Very 7 

high 

6 

5 

4 

3 

2 

Very 1 low Clarity of Conciseness Level of Accuracy of Noticeability Usefulness ofUsefulness of Timeliness of 

information of detail shown information of needed information information information 

information by USS information for planning for making a delivery 

decision UTM information properties 

Figure 7. UTM information properties during DAT testing in TCL3 shown by type of DAT test 

(n = 24–30). Note: Rating scale was 1–7, y-axis is longer to show SD. 

4.4 DAT 1: End-to-End UREP 

The DAT 1 test was to send UREPs (UAS operator reports) through the UTM system to be received 

by others in the vicinity (see Rios, et al, 2018, for more details). Test sites were to gather data on the 

update rate, the data quality, the appropriateness of the data elements, and the effectiveness of the 

Supplemental Data Service Provider (SDSP) architecture. Survey questions asked about messages 

sent and users’ thoughts about the information that UREPs provide (information exchange), while 

debrief prompts focused on the uses for, and automaticity of, the UREPs sent (Table 8). 

Table 8. DAT 1: Discussion Topics and their Notional Categorization 

into Information Themes 

Information Category Site 2 Site 4 Site 5 Site 6 

Information property 3 4 

Information meets user 2 3 3 1 

Operator status 

Methods 3 1 

Automation 1 3 

Concept 

17 

 

 

 

 

 

 

 

 

 

information about the local flying environment, such as noting flocks of birds or manned traffic in 

their flight area, and other UAS. Whereas one respondent said they would use UREPs to broadcast 

safety issues, another said that other means should be used for sending safety critical information as 

UREPs are only meant to share non-critical information. Users cited a number of additional features 

they would like to be offered by their USS UREP software: an ability to categorize the type of UREP 

sent, a tighter integration of UREPs in the USS software, and an alert to draw attention when a 

UREP arrives. (Note that some USS clients already have some of these features and every USS 

implemented the UREP functions differently.) When asked if they would like a way to send a return 

message, as this is not a function currently available, some crews declined, stating that UREPs are 

for information only, they are not intended to be a conversation channel. 

4.5 DAT 2: FIMS Failover 

The DAT 2 testing was to note the effects on operations and the UTM system in the event of a FIMS 

failure. Survey questions asked about operator awareness of the failure and concerns while FIMS 

was down (operator experience). However, most participants did not know FIMS had failed; only 

one in four confirmed that they knew when the system went down. Answers to subsequent questions 

were predictably, and perhaps artificially, benign due to the lack of awareness, with participants 

(very low n) reporting low levels of additional effort to maintain operations during FIMS failures, 

low levels of concern for safety, and good operational consistency while FIMS was down. 

4.6 DAT 3: USS Failover 

The DAT 3 testing was set up to fail all or part of the USS, note the effects through the UTM 

system, and to develop procedures for such an event. Survey questions asked about the impact of the 

failure on both the operator and the flight operations (operator experience), while debrief questions 

focused on the operator’s experience, procedures and alerting (Table 9). 

Table 9. DAT 3: Discussion Topics and their Notional Categorization 

into Information Themes 

Information Category Site 2 Site 4 Site 5 Site 6 

Information property 4 

Information meets user 2 1 

Operator status 4 3 

Methods 2 8 

Automation 1 4 

Concept 4 

4.6.1 Operator Experiences 

Crews reported that when their USS client was down for only a short amount of time, they did not 

have any issues when the system came back up. And if it was a limited fail, there were still some 

USS features available to them. But, when the USS client was down for longer, there were 

“surprises” as it came back on line, such as finding your UAS status to be rogue. Crews noted that 

the system does not know it has failed and is “catching up” as it comes back on line, that means they 

have to wait for information to repopulate also. 

19 

 

 

 

 

 

 

UAS operations under 55 pounds. For these crews, Part 107 was a way to keep flying outside the 

confines of UTM, with one participant noting that “operators don’t want to be restricted” when they 

feel there is no “reason” for it. 

4.6.3 Methods: Procedures 

Crews noted that, having executed DAT 3, they would like more definition about how to handle USS 

failures. They realized they needed to assess what was happening, what the impact of the failure 

was, what other tools they had available, and what they knew at that point in time about other 

operations in the air. Based on this assessment, they needed to decide about how to act. They felt 

that if this evaluation and option choice process was more formalized, it would have helped them 

react to the DAT 3 situation. Also, for those using a different client, what should they do on finding 

that another USS client operating in the same area is down? One test site discussed that procedures 

could be defined for different levels of USS failure—a short outage, versus a medium length outage, 

versus a long outage. For example, with a short outage, you may have an option to hover in place 

until the USS client comes back up, but for a long-term outage you may need to RTB or land in 

place. 

4.6.4 Information Exchange: Alerting 

Crews suggested that when a USS client fails, there should be some kind of alarm or alert to warn 

them that it is offline. Crews noted there were too few indications that their client had failed, as some 

parts of the display continued to update. There were also some concerns about the strain that would 

be placed on the human-communication system during a USS client failure, as users call client 

suppliers when they lose their display. 

Crews talked about the wider ramifications of a failure like a USS failure. If you have other tools 

that allow you to see the locations of other vehicles, i.e., some redundancy between systems, then the 

loss of your USS is not critical, however, if your USS client is your only source of detecting other 

aircraft, losing it is a critical failure and your choice of action would need to reflect that. For 

example, if the USS was providing a surveillance service, then when it goes down you lose your 

awareness of what is close to your airspace, and if you happened to have a vehicle transiting your 

area then you would lose awareness of what it is doing. At this point the threat is very different from 

a situation where your USS is not providing surveillance. Hence, threat assessments when a USS 

goes down can vary widely by situation. 

4.6.5 Automation 

Test sites did learn about the functionality of, and the graphical user interface (GUI) of, their USS 

clients through this exercise. While some found that their USS sent automatic RTB notices some 

number of minutes after failing, others found their vehicles were assigned a rogue status. One 

commenter was concerned that having one USS down may bring others down as the system keeps 

trying to report data to a USS that is not accepting data. Crews said that the only message that needs 

to be continually shared when a client is down are “rogue” messages, and all others could be 

suspended to reduce load on the system. Observations made as a result of using the USS GUI during 

a failure included that the USS status indicator (health) needs to be very clear, as does vehicle 

volume status. 

21 

 

 

 

 

 

 

 

8 

Very 7 high 

Site 3 Site 4 

6 nge

5 excha

lness 

Mean data 

usefu 4 

3 

2 

Very 1

low Data exchange usefulness Data exchange timeliness 

Data exchange properties 

Figure 10. UVIN/ UAS ID data exchange usefulness and timelines (n = 18–19). 

Note: Rating scale was 1–7; y-axis is longer to show SD. 

4.7.2 Operator: Communications 

Participants suggested that voice communications (over the radio) worked well in certain 

circumstances. They noted that if you were a UAS operator working where there are manned 

aircraft, a good way to ensure broad situation awareness among all parties would be to announce 

your intentions on the radio in the same way that pilots of small general (GA) aircraft do. To do this, 

all UAS operators would need to know how to follow aviation radio conventions and carry a 

handheld radio. The participants acknowledged that this may not be a scalable solution if there are 

hundreds of UAS in the air but in a local area with just a few operations this could be a workable 

solution. 

4.7.3 Concept 

One crew stated that they thought the concept was useful to enhance situation awareness for their 

operation to be able to identify a UFO in real-time. Crews noted that the UAS ID procedure will 

identify operators who are “clueless” but not those who are malicious. In general, crews reported 

that pilots will comply with the UAS ID procedures as they do not want to get into trouble, but 

again, those who have ill-intent will not care about this. One crew noted that the registered user for 

their vehicle, who was revealed during the UAS ID test, was someone who no longer worked in the 

group. They commented that managing and checking valid information attached to UAS ID could be 

a large task. 

4.7.4 Automation: Technical 

A number of technical issues were raised during debrief discussions: 

• Crews thought that when they were flying their vehicle away from their GCS, that the 

vehicle itself shielded the UVIN/UAS ID transmission. More testing is required to 

explore this. 

• Crews had some concerns that the ISN spectrum may not be broad enough for the 

usage that it will get for UAS ID transmission but thought that it worked well. 

23 


8 

Site 2 Site 4 Site 5 Site 6 

Very 7 

high 

6 

5 

4 

3 

2 

Very 1 

low 

0 

Mean rating 

 

 

 

 

 

 

 

Workload Timing Situation awareness Situation awareness 

of your operation of surrounding 

airspace 

User experience of USS-USS negotiation 

Figure 12. User experiences during the USS-to-USS negotiation process (n = 19–26). 

Note: Rating scale was 1–7; y-axis is longer to show SD. 

4.8.4 Method 

The methods for USS-to-USS negotiation need polishing. Crews suggested one area in particular is 

to set a limit so that negotiation is not occurring at the time of launch. Participants said firstly, that it 

is awkward and distracting to be negotiating during the take-off sequence and procedures should be 

set to avoid this. Secondly, if you have planned and set up a flight for weeks or months, it will be 

disruptive, frustrating and disappointing to have your plan negotiated away. They suggest there 

should be a clear window before you launch where someone else cannot request your planned 

volume. 

4.8.5 Automation 

The amount of automation employed for the DAT 5 tests differed among the test sites. One on-site 

NASA representative noted that human intervention was not required for the operations at one test 

site because of the high level of automation. At another test-site, the NASA representative observed 

that verbal communication was necessary for coordinating DAT 5 tests, and asked, “What if pilots or 

GCSs weren’t close enough to each other to verbally chat. Would that change a response to a request 

for clearance to fly in the same airspace?” (Appendix 8). 

4.8.6 Concept 

In their survey comments, many agreed that USS-to-USS negotiation was an interesting and useful 

exercise, although more complex than they had anticipated. The issue of other USS “policing” or 

“blocking” space was raised a number of times. Participants noted instances where they were unable 

to complete their testing as they wanted because they had been blocked by others, e.g., an interesting 

case with one USS policing other operations in the local USS network (LUN). The crew with this 

USS viewed that as a capability but others who were shut out did not: “We were conducting 

coordinated flights. With the not-involved USS sending a rejection, we had to cancel.” All 

commenters recommended that this should not be possible within UTM. 

26 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

– Procedures: When a USS failed, it was unclear how crews needed to react. 

Moving forward, test sites should be encouraged to have a set of procedures that 

specify what to do in the event of a USS failure. 

– Procedures: Crews found negotiation as they were trying to launch their vehicles 

distracting and workload intensive. Moving forward, it is suggested that the

negotiation end time be specified before a flight begins and/or before the volume

becomes active. 

– Gaming: When they found their airspace already in use, some crews switched to 

become a Part 107 flight to avoid delays, others set up large volumes to clear the 

airspace for themselves. Moving forward, rules need to be developed to prevent 

users gaming the system. 

– Trust: To negotiate successfully, users need to share an amount of information, 

which may vary depending on the conditions. Moving forward, an effort to 

specify the information that needs to be shared for productive negotiation under a 

variety of circumstances would be useful. 

• Interfaces for reading and generating messages need to be improved, including alerting 

and reducing clutter 

– Usability: USS client usability in general, and their available features, varied. 

Moving forward, useful steps could be to implement some of the suggestions 

crews had for improving their USS client displays (e.g., a UREP search function, 

that messages persist somewhere but do not clutter the main screen). 

– Alerting: Crews thought the system failure alerts were poor. Moving forward, 

there is a need to improve the alerting for system crash events. 

• Crews realized that maintaining an awareness of DAT messages and USS interactions is 

useful and could be important 

– SA: While general SA was reported to be good, crews recognized the need for good 

SA at all times, as they will need to draw on their own knowledge in the event of 

a system crash. 

– Function: As some USS came back after a (planned) system failure, crews 

discovered a number of unexpected features in their USS clients. Implication: It 

would seem useful to explore these corner cases, with an aim to make the 

functionality of USS recovery more user friendly. 

5. Concepts Tests 

The Concepts and Use Cases Research Transition Team (RTT) Working Group is responsible for the 

collaborative development and definition of the UTM concept in terms of overall principles and 

assumptions. Elements of the concept are conveyed through use cases that span the range of UTM 

operations and serve to identify information flows, highlight roles and responsibilities of 

stakeholders, and provide an understanding of envisioned operating environments. For the TCL3 

flight tests, there were a total of five concept tests that were conducted at multiple test sites and were 

referred to as CON tests. These tests were designed to exercise key areas of the TCL3 concept and 

provide input to other UTM RTT Working Groups and programs as required. 

The CON 1 test investigated the planning considerations and technologies associated with a BVLOS 

landing, takeoff, and return to base. Test sites were encouraged to vary the locations of their BVLOS 

28 
 

 

 

 

 

 

 

 

 

operations to provide insight into a wider range of applicable scenarios. The Contingency Initiation 

test (CON 2) had test sites coordinate a planned emergency that necessitated activating a prepared 

contingency plan. Examples of possible contingencies were given as a “ditch” at a known location, a 

return to base, or an automated loiter, and test sites were again encouraged to develop several 

configurations of this test. CON 3 looked into the usability of a UTM public portal. Test sites 

demonstrated the safety of flying Multiple TCL2/3 Operations for a sustained period in CON 4, in 

the same airspace, demonstrating a more realistic dynamic environment. These were interacting 

operations with distinct scenarios (e.g., multiple deliveries, traffic monitoring, infrastructure 

inspection, real estate photography) to be flown for a few hours. Lastly, the CON 5 tests explored the 

FIMS/USS interaction when a vehicle headed towards controlled or unauthorized airspace. 

Five of the six test sites were awarded one or more CON tests, with one site being awarded all five 

CON tests. Note that the CON 3 public portal test results will be analyzed in a separate effort from 

the current document. From here onwards, CON3 will be excluded from the counts and tallies 

reported. In total, 14 CON tests were awarded and these awards were distributed so that each CON 

test was performed by either three or four test sites. Test sites conducted these 14 tests over 15 

calendar days, across three months, often performing them in parallel with other tests. 

5.1 Method 

5.1.1 Surveys 

There were 58 questions in the CON survey, with the questions preselected to show only between 15 

and 29 of these questions depending on the test, e.g., those completing CON 4 saw 15 questions— 

nine general and six specific to the CON 4 test—as well as information questions to determine 

participants’ role and test site. Those who answered questions about CON 2 received 29 items, the 

same nine general items and 20 CON 2-specific questions. Many items asked the participant to rate 

their answers on a 1–7 scale, where 7 was high, or positive, and 1 was low, or negative. Other 

question types were multiple choice and free-response. Sixty-one surveys were started in total across 

the five sites, all of which had some data entered but not all were complete. The greatest number of 

surveys started at one site was 19 and the fewest was eight, but note that although one site undertook 

all five CON tests, another only undertook one. 

5.1.2 Debrief 

There were 16 debriefs that focused on CON tests hosted across the five test sites. Twelve were 

specific debriefs focused on one CON test while the other four touched on more than one (see 

Appendix 10). Numbers of discussion prompts varied as researchers wanted crews to discuss and 

explore the topics within the time available. Prompts focused on operator situation awareness, the 

way the UTM system added value to the test flights, the perspective crews had gained from the tests 

and technical issues of concern. 

5.2 CON Metadata 

CON tests accounted for the most operations for TCL3 testing with 401 operations submitted to 

UTM. Of these, 27 originated from the NASA lab for collaboration with the test sites, while 374 

originated from the test sites themselves. CON 1 had 13 operations submitted by the test sites, CON2 

had 22, CON4 had 311, and CON5 had 28. As shown in Table 12, there are gaps in the data where it 

is likely that flights that were participating in certain CON tests were simultaneously serving a role 

in a parallel test but were not noted as such in the data. 

29 

 

 

 

 

 

 

 

 

 

5.3.2 Information Properties 

Participants were invited to rate eight properties of the UTM information they received through their 

USS (Figure 14). On average, participants rated all the UTM information they received as “good” 

with means varying from 3.7 for noticeability of needed information to 4.8 for accuracy of 

information. Participants gave the highest ratings (on average) after CON 1 and the lowest after 

CON 5, where there was a FIMS/USS interaction. 

8 

CON1 CON 2 CON 4 CON 5 

Very 

high 

7 

6 

1 

2 

3 

4 

5

Mean rating 

Very 

low Clarity of Conciseness Level of Accuracy of Noticeability Usefulness Usefulness Timeliness 

information of detail shown information of needed of of of 

information by USS information information information information 

for planning for making a delivery 

decision 

UTM information properties 

Figure 14. UTM information properties during CON testing in TCL3 shown by type 

of CON test (n = 33–39). Note: Rating scale was 1–7; y-axis is longer to 

show SD. 

Participants were asked about workload after every CON test, but the question was tailored to a 

specific time, so each of the bars in Figure 15 represent slightly different questions. On average, 

participants rated their workload as “low” to “moderate” during the CON flight tests, with CON 4 

having the highest ratings, when they were asked about workload during first and last flight sessions 

of the day. Crews’ lowest workload ratings were during CON 1, when they were asked about when 

they were landing/taking off BVLOS. 

31 
 

 

 

 

 

 

Mean workload 

Very 7 

high 6 

5 

4 

3 

2 

Very 1

low CON 1 CON 2 CON 4 CON 5 

CON test 

Figure 15. Operator workload during CON testing in TCL3 shown by type 

of CON test (n = 3–23). 

5.4 CON 1: Take-off and Landing while BVLOS 

The CON 1 testing asked crews to operate their vehicles beyond VLOS to land and take off 

remotely. Two of the three test sites arranged for their vehicle to launch within LOS, fly to a remote 

location, land, take off, and return to base. Test Site 3 operated their vehicle remotely for the 

duration of the flight and added an indoors/outdoors component to their CON 1 flight tests. Survey 

questions asked about users’ situation awareness during these maneuvers and users’ thoughts about 

the safety of their operations (operator experience and concept), while debrief prompts focused on 

situation awareness, risk assessment, and operation of UTM (Table 13). As intended, testing for the 

CON 1 TCL3 use case contributed to understanding of other non-CON tests and the general TCL3 

UTM concept. For example, at one test site, an interior BVLOS takeoff and landing also touched 

upon strategically using multiple communications links to maintain command and control of the 

vehicle when there were physical obstructions blocking the primary signal, similar to CNS testing. 

Table 13. CON 1: Discussion Topics and their Notional 

Categorization into Information Themes 

Information Category Site 1 Site 3 Site 6 

Information property 3 

Information meets user 6 

Operator status 6 

Methods 1 

Automation 7 

Concept 4 

32 


 

 

 

 

 

 

like if they wanted to run synchronized flights. Participants noted that if crews shared their flight 

plans and one was flying a longer-distance leg, the volume could either close behind it, leaving the 

space available to others, or the space could stay open and a second vehicle could follow exactly the 

same flight path, just a few minutes behind the first. 

Suggestions for new UTM features: Crews suggested a number of improvements to the 

information they received through their USS client. Note that these features were currently 

unavailable to participants through the particular USS client that they were using, but many of these 

features are available through other USS clients. 

• Crews wanted to be able to share a GCS launch area between multiple vehicles, that is 

launch one vehicle after another from the same pad. 

• In the case of overlapping volumes, when you are the second volume in line, 

participants discussed having a feature that would compare the parameters of your 

volume with the other accepted volume and would give you feedback on how long you 

will have to wait before the area is cleared and your volume could be accepted. 

• Crews felt that the UTM system should be able to take holds or delays into account and 

move all the schedules into the future by the right number of minutes. The example 

given was if one crew has a five-minute hold on the ground, UTM should estimate, 

based on the submitted flight speed or sensed speed of the first vehicle, how long the 

flight is likely to take and then tell other operators how long they might have to wait. 

• Crews noted that they would like a “clearer” set of information from the UTM system— 

something close to a countdown—where they would receive a message telling them 

how long they needed to wait after a previous flight launched before they could launch, 

then receive a notification when the first flight exited the launch volume, and a 

notification that they are cleared to launch. These ideas describe a system that takes 

something more like an air traffic control role, where the system assesses and assures 

clear airspace, which is not how the current UTM system is designed to operate. 

5.5 CON 2: Contingency Initiation 

The CON 2 test had test sites coordinate a planned emergency that necessitated activating a 

prepared contingency plan. Options included an incursion of one vehicle into the volume of 

another, or an “event” that required a vehicle to make an emergency landing. Two of the three test 

sites chose to have one vehicle breach the volume of another. The third test site set up an “event” 

where their vehicle had to land in place. Survey questions asked about users’ situation awareness 

during these maneuvers, their decision making and their reactions to their contingency plans, while 

debrief prompts focused on situation awareness, information that is useful, and risk assessment 

(Table 14). The contingency scenarios demonstrated in CON 2 also relate to other tests from TCL3, 

such as the general goal of the Sense and Avoid RTT to investigate conflict mitigations, and 

specifically the off-nominal conditions tested in SAA 5. From the UTM concept perspective, 

NASA is able to specify use cases that illuminate whether the current infrastructure and capabilities 

meet the needs of the user. 

5.5.1 Information Desired 

Participants listed many items of information that they would like to have when there is a vehicle in 

their vicinity that is having a contingency event. These items included being told the location of the 

vehicle in trouble, also its position, altitude, heading, velocity, flight trend, position history trail, and 

35 


 

 

 

 

 

 

 

manual responses did so because every contingency event is unique and needs a specific response. 

Those arguing for fully automated responses noted that the PIC will be too busy to send messages to 

other crews and local entities during an emergency situation. 

Supporting the comments of those who said contingency events are unique, debriefs explored that 

crews were creative in the ways they reacted to events. One crew, who was operating the vehicle 

with the emergency, designed an RTB route and then built a TFR around it, essentially putting a 

volume around their vehicle for the RTB transit. They discussed the advantages of doing this; flying 

inside a TFR helps to flag the flight to other users. 

The “innocent bystander” crews, who had another vehicle breach their airspace, said they would like 

the USS to tell them when that breaching vehicle (or vehicle that has an emergency and is transiting) 

has left their airspace, because they would like to know that it is safe to continue their mission at that 

point. One crew said that although their goal was to get their vehicle safely on the ground, they 

waited for the vehicle-in-trouble to land before they stopped their loiter, just to be sure that they 

would avoid a conflict. 

5.5.5 Procedures: Emergency 

In the case where there is an emergency, rather than have a TFR that requires operations to leave the 

airspace, crews discussed putting procedures in place where they make way for the emergency flight, 

“pull to the side of the road” to allow the flight to pass, but do not have to exit. From the other side 

of the problem, as an emergency vehicle, it would be good to know that the users of the airspace 

were being compliant and had vacated the space that you are now working in. The emergency 

responder would want to know that previous volumes were being closed and vehicles are down. 

5.5.6 Procedures: Gaming the System 

In general discussion, participants talked about some ways that bad actors could game the UTM 

system. They discussed that if you went rogue during the contingency event, this status then gives 

you priority over others, with the assumption that you are a disabled flight and you need direct routes 

to RTB. However, this creates a loophole that operators could take advantage of. If they submit a 

very small volume but then fly their mission in rogue status, under the contingency management 

rules this would give them a priority status over other vehicles to essentially bump accepted users 

out of their space, even though they were not in distress. 

5.6 CON 4: Multiple Operations for a Sustained Period 

The CON 4 test asked crews to fly their vehicles simultaneously for extended periods of time – two 

or three hours. Many test sites used these extended flight days to fly other tests concurrently. Survey 

questions asked about users’ situation awareness and workload over time, while debrief discussions 

covered procedures, issues with reserving airspace, and alerting (Table 15). 

38 


 

 

 

 

 

 

 

 

 

 

 

want crews they know and trust to fly in the altitude above them5. It is easier to book the whole 

vertical space to ensure that no-one flies over you. Examples of crews booking large volumes 

include: one USS Op, who said his crew’s mission was to map the entire area, so he submitted a 

volume early that took over the whole test area, blocking everyone else from submitting. He said he 

felt bad because he knew he was stopping everyone else from flying, so the crew landed after their 

first flight and released the airspace but then they had to wait four hours to fly again. He said that in 

a real setting, the crew would not have done that and would have completed all their flights at one 

time, even if it was tying up the whole airspace. Another crew, who was on the field, described how 

they initially could not get a volume (because another crew had reserved the airspace they wanted to 

fly in). When the first crew had finished, the second crew reserved a huge airspace so now they 

could fly un-interrupted, but said they amended it to a series of smaller volumes, as they looked at 

the airspace usage for the rest of the day and realized that doing this would delay other groups in the 

same way they had been delayed. Instead, they organized who would use which space so everyone 

could fly simultaneously. A third crew felt that when their flights are for revenue, they would not be 

able to wait for another group who has booked the whole airspace where they need to fly. They 

concluded that, in this case, they would simply file to fly as Part 107 to make sure that they could 

complete their flight on time. 

Participants stated that if the UTM system wants users to be as efficient as possible, it will need to 

provide incentives to users to construct efficient volumes (and possibly penalties for taking 

considerably more space than needed). Alternatively, the UTM system could help you to be 

efficient. Crews discussed that if you could put your flight path into UTM and have the system build 

a buffer around it, then that would enable efficient use of space, but the current system makes that 

impossible because it only accepts a limited number of points (10)6. 

In addition to issues of other crews booking large volumes and shutting them out of the airspace, 

crews told researchers that they had put dummy data into all the PII fields in their USS client. This 

meant when everyone at the test site began reserving airspace and their own reservations were 

rejected, they could not contact other teams to negotiate when the airspace would be free or how to 

share it. Crews complained they had to submit volumes using trial and error until they happened on 

an area that was accepted. One crew said they reduced the size of their volume until it was accepted, 

other groups used the site radio channel or personal phone contacts, i.e., pre-existing personal 

knowledge of other teams in the test, to negotiate verbally with others to carve out some airspace to 

work in. 

5.6.5 Automation: Messaging and Alerting 

Crews differed on their opinions of UTM alerting. One crew said they did not want too many alerts, 

while others said they would like warnings about nearing the edge of their volume to go directly to 

the vehicle. They suggested there could be automation that would allow the vehicle to interpret the 

message and speed up or slow down to address the issue. 

Crews debated how flying their vehicle manually might interact with a fully-autonomous vehicle. 

They were not sure they could negotiate with an autonomous vehicle and suggested that there should 

5 Also see section on faulty altitude readings. This is likely to add to crews’ reluctance to let others fly 

above them. 

6 At the time of TCL3, the system only accepted 10 volumes, but this issue has since been solved; 

users can now have up to 250 volumes per flight geography. 

41 
 

 

 

 

 

 

 

be clearly defined “rules of the road” and either the manual or automated system should have 

priority. Others disagreed that priority should be based on level of automation, saying that every 

vehicle should have equal priority. 

Participants suggested that USS clients should output the same messages as each other and these 

should be interpretable by whatever or whomever is receiving them (manual or automated). They 

stressed that every user should receive the same information and have equal chances to act on it. 

USS clients could display the information in different ways to suit the person/automation reading the 

messages but the base information going to all clients should be the same. That said, crews thought 

contingency messages should be automated rather than manually sent. Pilots noted that they did not 

have time to compose and send manual contingency messages in real time. PIC should not have to 

announce an emergency state to other crews in the area, that function should be automated. 

5.6.6 Automation: Rogue 

One USS operator said he would like to receive information from the USS about how close his 

vehicle is to the compliance boundaries. If he had this information, he would know whether he 

needed to adjust his vehicle to ensure that it stays well within its boundaries. He suggested it would 

also enable him to reduce his time allocated for different volumes. Others agreed for a different 

reason, saying that you do not get a lot of warning that you are going rogue and then you do not have 

time to react, and that more information about being in compliance might help you to avoid 

becoming rogue. 

The USS designers informed researchers that they can assign whatever state that they wished to 

flights, that they did not have to assign rogue status because the definition of rogue used by NASA 

only applies to the NUSS. In some USS clients, when a vehicle went rogue, the system built a series 

of tiled volumes. This kept the rogue vehicle in a volume but crews wondered about the UTM state 

that they were in at this point, and about what happened to the other volumes in the airspace that 

they may have been overcutting. It seems that the new volume tiles were assigned higher priority 

than the original volume and so the vehicle went from rogue to priority status. 

5.6.7 Automation: Safety 

Eighty three percent of respondents reported that they had no safety concerns about the safety of 

their vehicle during the test. Of the four respondents who did have concerns, one said it was because 

their vehicle was only sending intermittent UTM updates, another had concerns about altitudes, a 

third had to take manual control of their UAS, and the fourth said that using UTM is a task in itself, 

which made it distracting and it took up time that should have been used to focus on flight control. 

Crews debated whether UTM needs to be designed to accommodate airspace users who are not 

going to participate in UTM (Part 107 and others). They emphasized that there will always be 

vehicles in the airspace that are not following UTM, for example, crop dusters who may not want to 

install equipment so UAS can detect them. Others suggested that there should be incentives to 

encourage UAS crews and other airspace users to participate in UTM. This led to a discussion that 

UTM can be dangerous because it gives you a false sense of security that you are alone in your 

volume but, in reality, there could be other vehicles in that space that are not in UTM and that will 

not be shown on any of your displays. Discussions like this suggest USS developers might be 

encouraged to maximize transparency of the system and the use of detect and avoid (DAA) tools. 

42 
 

 

 

 

 

 

 

 

 

5.6.8 Automation: Using USS and UTM Information 

Crews liked the USS client information that showed them volumes and said this helped them to stay 

in their volume. Some crews (depending on the USS client) were only able to see other flights that 

were using the same USS on their USS client display7. Not all crews were aware of this and that 

those using a different client would not be shown to them. They were confused at not being able to 

see the volumes for all operators in the area and commented that it made their task far more difficult. 

One USS operator said that his map was easy to use until he had to negotiate and then he could not 

see his plan to help him in that negotiation. Another USS operator said that managing only two 

flights was not an issue but he could imagine that trying to manage a dozen flights would be very 

difficult because the interface was not efficient. Another team commented that they would have 

liked to have their volume shrink dynamically as they covered the space they needed to survey, 

opening the area behind them up for others. Crews discussed that, with their USS client, that type of 

dynamic re-planning would have been very difficult to do. 

Participants discussed the real-time reservation function of UTM. They complained about not being 

able to queue multiple volumes in advance and having to work in real time. They noted that the 

ability to reserve a space up to 24 hours in advance would have made their pre-flight roles much 

easier8. USS Operators complained that the process to manage multiple flights is cumbersome, they 

had to submit in one order, activate in another, and all while crews were waiting to fly. 

One USS Operator reported that planning flights when there were many crews trying to submit 

volumes and plan was also challenging. He said his screen became cluttered with all the plans that 

were being submitted and, because he did not know the times of the other plans, it became difficult 

to make effective decisions. He had to submit plans and “see what happened” or waited until it 

looked like others were landing and tried to resubmit right at that point, hoping to get their space. He 

suggested a filter that would allow the USS Op to filter submissions by start time, but this would 

require all UTM users to share their volume start times. 

Differences between clients were extensive: some users saw contingency messages, while others did 

not and had to integrate the information manually, because this functionality was not built in their 

client. Crews thought conflict alerting in the USS client would be useful, and automated messages 

from the USS would be helpful. They did not think the GUFI should be something that they should 

have to track. 

Additionally, crews were still learning the way their USS clients operated. One crew, simulating a 

vehicle with an emergency, flew out of their volume to create an alert through the system. They were 

surprised when their USS created a new volume with a straight flight path to RTB. Users came 

across a couple of issues, or bugs, using their USS client. One group could not modify their start 

time to an earlier time and another group had an arrow that appeared as a target. A second group 

thought there was 30 seconds of lag in their system, which they said would make it of no use. 

Another USS operator could see his TFR, but was unable to select a new plan to stay in 

conformance, so their flight went non-conforming and then rogue while he was waiting for a 

volume, that would have kept him in conformance, to come live. 

7 Again, this function/ capability is USS dependent. 

8 Reservations in the future were possible through the UTM system but the USS developer had to 

implement this functionality in their client also for it to be available to crews. 

43 


 

 

 

 

 

 

 

 

 

 

 

they received a proximity warning and an alert as they crossed the boundary. Another said this was 

“more than enough” information to receive. Count 

6 

USS client GCS display another crew member 

5 

4 

3 

2 

1 

0 

Approaching Breaching Nearing controlled Breaching 

operational operational volume airspace (proximity controlled airspace 

boundary (protected alert) (breach) 

(conformance boundary) 

boundary) Types of alert 

Figure 20. Types of alert and sources of this information (n = 11). 

Alerting Format: Some participants appreciated that the audio-alert they received that gave them 

range and bearing of the intruding rogue vehicle. Others preferred a graphical representation and 

thought the PIC should consult a display. 

Alerting Triggers: Participants were asked about the most appropriate way to scope and trigger 

alerting. Of the respondents (n = 12), 47% thought the most appropriate way to set alerts would be 

distance-based. Participants varied in how far they thought this distance should be, from 50m to 

926m (0.026nmi to 0.5nmi). 29% thought a time-based alerting system would be the most 

appropriate, estimating a 10 second warning would be best. In debriefs, some said it should be sent 

when the vehicle is 50–75m from the boundary. Others said the warning should be based on vehicle 

velocity and the warning should be issued when the vehicle has 30 seconds before it will meet the 

boundary. Crews definitely wanted warnings if they were flying BLVOS but argued that when they 

would receive the warning depended on the type of boundary that was about to be crossed. They 

would like a warning for a TFR boundary but if it is another operator’s volume then they did not 

think they would need a warning. Twenty-three percent of the group thought they should not receive 

alerts about breaching a restricted airspace because at that point the vehicle is already rogue (outside 

its volume), which would have already triggered an alert, and the crew should be doing everything 

they can to RTB or fly back into their volume. 

Poor Alerting: There were two aspects of the alerting that participants felt could have better 

usability. The first was the short elapsed-time between being non-conforming and rogue. Participants 

commented that having only two seconds (some said 30 sec) to react and rectify the issue was not 

long enough, so they inevitably went rogue. In one case, the vehicle went non-conforming in its 

descent volume and, due to the descent taking longer than 30 seconds, the vehicle was rogue by the 

time it landed. Users also complained that once they had breached a volume or there was a pop-up 

TFR, their alerts repeated every few seconds. A number of crews said they did not need repeated 

messages that they were rogue, nor about the TFR, because they knew after the first alert. They 

46 
 

 

 

 

 

 

 

 

 

 

 

would have liked a way to acknowledge, and then silence, that alert. Even without an 

acknowledgement, participants did not want it to repeat every few seconds and suggested a repeat 

cycle longer than 30 seconds would be better. 

5.7.5 Warnings 

Crews debated about who should be warned or sent a notification when one vehicle breaches the 

volume of another. Some participants advocated that only the two parties involved should know, and 

others went further to suggest that if UTM calculated that the vehicles were not on conflicting 

courses then the reserver of the volume (in conformance) did not need to be notified. Others argued 

back that the reserver of the airspace should always receive a message about activity in their airspace 

because if the intruder does not take action, the vehicles could quickly be on a collision course. 

Another group argued that everyone in the vicinity should receive the message as this kind of “party 

line” information can be useful9

. 

Participants compared this situation to one in which a TFR is created near your volume while you 

are flying. Respondents argued that a notification about the TFR should go out to every operation 

that is now inside the TFR and along its borders. However, if someone breaches the TFR once it is in 

place, crews argued that only the intruder and the vehicles operating in the TFR should be notified. 

Associated with this operator awareness debate is a UTM concept debate. Currently, vehicles 

operating under one USS system cannot see the vehicles operating under another USS system, even 

though both systems may have a LUN that encompasses the same physical geographic area. Because 

of this, it was argued that more operators rather than fewer should be notified about volume breaches 

because there is not a way for them to notice this for themselves on their display if the intruder is 

using a different client (note that this situation is possible but is USS client-dependent). These same 

crews noted that, strangely, the general public would be able to see all vehicles through a public 

portal and so crews may have to run a public portal as well as a USS client if they would like to see 

all traffic in their LUN. 

5.7.6 Usability of Warnings 

Along with debating about who should receive messages regarding intruders into airspace, crews 

listed information and parameters that would improve the usability of the messages. One team 

thought that messages should contain a potential impact assessment. Another said that the severity of 

the intrusion should be color coded by comparing its flight path relative to the approved operator in 

the volume, e.g., red to indicate that the intruder is heading, within some band of degrees, towards 

the other vehicle, and yellow to indicate the intruder is not going to fly so close to the other operator. 

5.7.7 Temporary Flight Restrictions 

Crews discussed TFRs at length. The groups felt that notifications for popup TFRs should be 

through the UTM system, probably using a graphical feature to draw the area, as crews who only 

received text proximity warnings but could not see the boundary had trouble developing awareness 

of the situation. One participant stressed that they needed to know exactly where and when a TFR 

was going to be in order to assess how it would affect their operation. 

Participants considered a number of the rules that govern UTM and how these impacted them in 

CON flight testing. One was the length of time that crews should be given to vacate their volume 

9 See the section below this section about how wide the broadcast of a TFR should be. 

47 
 

 

 

 

 

 

 

 

 

 

when a TFR is put up over it. One USS client gave crews two minutes to vacate while another gave 

crews five. Crews debated how even five minutes may not be enough to complete a mission, if you 

were flying a long way BVLOS and the TFR was large. Crews liked having a five-minute warning 

that a TFR was going to go active, but thought a graphical depiction of the TFR would be 

additionally useful to show which areas of your volumes are going to be affected. 

With respect to long distance flights, i.e., if you are flying a distant BVLOS and the TFR pops up 

over your return route, meaning you may not be able to get back to and across that airspace during 

the five-minute limit, participants noted there needs to be a contingency for cases like this to allow 

them to RTB within conformance. One participant suggested that, since non-priority volumes have a 

time limit10, if you are in the air the TFR should not apply to you. As many emergency responses are 

time critical, it is unlikely that First Responders would be prepared to wait for deliveries to complete. 

Crews debated how far/close from their volumes that they would want to be notified about upcoming 

TFRs. One participant said they should be notified of any TFR within one or two miles of their 

volume when they were VLOS, but when you are BLVOS that the limits should be different. 

Another said that they would like to be notified of a TFR that is within a mile radius of any one of 

their flight path waypoints. All agreed that those which are “30-plus” miles away from your volume 

are not of interest (or do not necessitate notification) but could not decide on the maximum distance 

at which a TFR should be shown. 

Crews thought it would be good to be notified how long a TFR is going to be in place. If you are 

flying a long mission, you may be able to work in one area of your volume until the TFR expires and 

then go back to working in the area that had been restricted. 

Crews spoke about whether TFRs should be negotiable like volumes for non-priority flights. 

Although some crews thought that you should be able to negotiate, others said that only approved 

organizations (e.g., Homeland Security, the emergency services), should be allowed to put up TFRs 

and so you would have to trust that the TFR is necessary, at which point negotiation becomes moot. 

Others thought there should be a way for crews to request permission to enter a TFR, with the host 

entity having the ability to approve or deny the request. 

5.7.8 Procedures: TFR 

Crews voiced the need for more defined procedures for handling a TFR. From their GCS positions, 

some crews were unaware that there was a TFR. Others, who did see the alert, said it did not contain 

enough information for them to make decisions about it, for example, the message itself did not 

contain enough information about the nature of the TFR to allow them to determine if it was safetyrelated. They knew that they had to exit the TFR, unless they were coordinated with the organization 

setting it, but were uncertain whether they had to take the shortest route out or could exit along their 

flight path. 

5.7.9 Emergency Situations: TFR 

Participants discussed the use of TFRs in emergency situations, suggesting that the emergency event 

coordinator/manager (Airboss) would put up a TFR around an airspace that the emergency services 

need to work in. Participants considered who within that space would need to know about the 

activities and, in general, agreed that everyone within that space, manned and unmanned operators, 

10 For TCL3, this time limit was 10,800 seconds, which is three hours. 

48 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Functions: One operator noted that a UTM function that assists with airspace design would be 

helpful—if you apply for a volume that is rejected because it is too big or overlapping another space, 

it would be useful to have the system tell you where or how to modify your volume (make it smaller 

or cut out a corner, etc.) in order to be accepted. 

UTM issues: It should be noted that USS clients displayed TFRs and rogue messages in different 

ways. For some participants, their launch volume was not quite correct and they received nonconformance or rogue warnings as they took off. This was annoying but it also marred their opinions 

of the valid TFR alerting. 

Participants found the repeated messages through their USS client irritating, arguing that they 

understood the message on its first announcement. They also found that some messages did not 

contain useful information. For example, the message through one USS client was “long” but did not 

contain information about where the breaching vehicle was. 

One USS operator could not see volumes of other flights, he could only see where other volumes 

overlapped areas in his volume, so he had to flip between earlier and later versions of his volume 

plan and compare what areas were missing from his original submission to work out how he had to 

amend it to a point where it would be accepted. 

5.8 CON Tests Summary 

CON flight tests asked test sites to generate back-stories for missions, and fly BVLOS or for long 

periods. This exercised the UTM system in a different way from other tests, revealing a number of 

issues with procedures, rules-of-play and alerting, as well as the USS client/UTM usability issues 

that were common to all tests. Some of the main issues were: 

• Procedures for reservations and negotiations could be more formalized. Incentives need 

to be given to promote efficiency, and feedback about the status of applications needs 

to be improved. 

– Function: Crews discussed issues they had with UTM volume buffers and with 

the UTM time element. Moving forward, they requested adding some flexibility 

to the UTM process (e.g., buffer options). 

– Design: Crews acknowledged that some USS will develop to be as automated as 

possible, while others will be much more manual. Implication: The same 

messages for an event may need to be displayed by all USS clients. 

– Procedures: Crews discussed both changing UTM status during an emergency and 

possible actions to maintain an ‘active’ status. Implications are that specifying 

procedures or rules for how operators should act under emergency conditions 

could assist. 

– Gaming: Operators gain if they over-reserve airspace but that puts others at a 

disadvantage. Other operators helped themselves by utilizing emergency status 

when they did not have an emergency and, again, that puts others at a 

disadvantage. Moving forward, rules should be set to prevent gaming the system. 

– SA: While general SA was reported as good, crews recognized the need for good 

SA at all times, especially during indoor-outdoor maneuvers and emergencies. 

Moving forward, crews listed information they would want provided about 

emergency operations. 

50 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

– Privacy: These tests raised the question of supplying personal information—in 

CON tests, crews found they needed to contact each other. Moving forward may 

require specifying the level of PII that needs to be provided with emergency 

handling in mind. 

• Crews differed in their opinions of UTM alerting depending on the situation. They 

wanted to be alerted as soon as possible to events that could affect their flight or 

airspace but at the same time did not want to be distracted by repeated alerts or 

warnings that were “obvious”. 

– Alerting a): Crews were concerned both about being alerted in a timely manner 

but not being distracted by alerting. Moving forward, there is a need to establish a 

sweet spot, listing what should be alerted and how often that alert should repeat. 

– Message effectiveness: Users compared their USS displays and found they 

received different messages at different times. Moving forward, the same 

messages for the same events may need to be displayed by all USS clients. 

– Alerting b): Crews debated who should receive warnings when there is an intruder 

into another operator’s volume. Moving forward, this indicates that procedures or 

guidelines for which operators should receive intruder warnings should be scoped. 

– Distractions: Crews commented that they passed too quickly from the UTM 

active status through non-conforming to rogue, and having to respond to the 

rogue state created a distraction. Moving forward, a review of the UTM rogue 

parameters is suggested. 

– Usability: USS screens became cluttered by various items of information where 

there was either too much or repeated information. Implication: Encourage 

developers to review their USS client GUIs to keep information available but not 

overlay other information on the main screen. 

• Participants did not question the need for TFRs but the rules and procedures governing 

them were unclear. Rules for when TFRs will be generated need to be clarified, as do exit 

times and procedures for exiting, and who, if anyone, has clearance to remain in a TFR. 

– Procedures: Crews were unsure of what they were meant to do when a TFR was 

set over the airspace they were using. Moving forward, procedures or rules may 

need to be specified for how operators should act under a TFR. 

– Public and hobby users: These tests led to more broad discussions including that 

of having users outside the UTM system and that this is problematic for SAA. 

Implication: Users suggested offering hobbyists incentives to encourage full 

participation in UTM. 

6. Sense and Avoid Tests 

The SAA tests focused on researching and defining technological solutions that will support 

successful sensing of conflicts in different situations. The sensor technologies employed in these 

tests included Dedicated Short Range Communications (DSRC) radios, ADS-B, and airborne and 

ground-based radars. Scripted situations included conflicts between two UAS, UAS and manned 

aircraft, and UAS and a ground vehicle. The interaction of these technologies as well as off-nominal 

procedures, including priority operations, were also studied. 

51 
 

 

 

 

 

 

 

 

 

 

 

 

 

For TCL3, six different tests were defined to explore these areas of interest. The first four SAA tests 

(SAA 1–SAA 4) evaluated the effectiveness of detecting and resolving airborne tactical conflicts. 

These four tests involved two airborne vehicles executing planned “conflict” events while gathering 

data on sensor and operator performance. Test sites were encouraged to fly a variety of conflicting 

live or virtual flight profiles and incorporate secondary conflicts and nearby airspace or ground 

constraints. Additionally, two of these four tests investigated cooperative technologies for conflict 

mitigation; SAA1 with a UAS-UAS interaction and DSRC, and SAA 2 with a UAS-manned aircraft 

interaction using ADS-B In for the UAS, and ADS-B Out for the manned aircraft. The other two 

tests investigated using airborne radar (SAA 3) and ground-based radar (SAA 4) as non-cooperative 

technologies for conflict mitigation in UAS-manned aircraft interactions. 

SAA also looked at the interoperability between each mitigation technology in a system level 

assessment under nominal and off-nominal conditions (SAA 5). In addition to evaluating the 

technologies used, test sites were asked to develop procedures for off-nominal conditions and 

investigate how priority operations might change those procedures. The SAA 6 test aimed to 

demonstrate the interoperability and remote identification of a UAS with automobiles using DSRC 

cooperative technology. As DSRC technology was a primary target for two of these tests, the results 

of SAA 1 and SAA 6 are presented jointly in this report. 

Three of the six test sites were awarded two or more SAA tests. In total, ten SAA tests were 

awarded, and these awards were distributed so that SAA tests 2–5 were performed by two test sites. 

Test sites conducted these ten tests over 27 calendar days, across three months. 

6.1 Method 

6.1.1 Surveys 

There were 55 questions in the SAA survey, with the questions preselected to show only between 23 

and 26 of these questions depending on the test, e.g., those completing SAA 6 saw 26 questions— 

nine general and 17 specific to the SAA 6 test—as well as a couple of information questions. Those 

who answered questions about SAA 5 received 24 items, the same nine general items and 15 SAA 5-

specific questions. Eighteen surveys were started in total across the three sites. Many items asked the 

participant to rate their answers on a 1–7 scale, where 7 was high, or positive, and 1 was low, or 

negative. Other question types were multiple choice and free-response. All 18 surveys had some data 

entered but not all were complete. The greatest number of surveys started at one site was 13 and the 

fewest was two, but note that whereas two sites completed four SAA tests, one site only undertook 

two. 

Although two test sites performed flight tests using ADS-B (SAA 2, with a total of 12 recorded data 

collection flights); two test sites performed flight tests using airborne sensors (SAA 3, with a total of 

seven recorded data collection flights); and two test sites performed flight tests using ground-based 

radar (SAA 4, with a total of nine recorded data collection flights), only the general section of the 

surveys were filled in for these tests. 

6.1.2 Debriefs 

Four debriefs were conducted which focused on SAA tests which were hosted across the three test 

sites. Three were specific debriefs focused on one SAA test while two focused on two SAA tests 

(see Appendix 12). Numbers of discussion prompts varied as researchers wanted crews to discuss 

and explore the topics within the time available (Table 17). Prompts focused on the topics of 

52 
 

 

 

 

 

operator situation awareness, information required and obtained, the need for sense and avoid 

technology, and technical issues of concern. SAA directly related to SA tools and displays, so these 

topics were central to discussion in debriefs, also. 

Table 17. SAA: Discussion Topics and their Notional Categorization into 

Information Themes 

Site 6 Site 1 Site 6 Site 1 

SAA Test Discussed SAA 1 & 6 SAA 2 SAA 3 SAA 4 

Information Category 

Information property 9 17 21 15 

Information meets user 17 10 15 6 

Operator status 11 43 4 5 

Methods 3 10 3 9 

Automation 17 13 15 13 

Concept 21 0 0 1 

6.2 SAA Metadata 

The SAA tests had the fewest submitted operations at 41, but the total number of flight activities is 

likely to be larger because not all flights were intended to be cooperative and connected to UTM as 

per the specific test requirements. Additionally, fewer sites were awarded these tests to conduct for 

TCL3. See Table 18 for a further breakdown of submitted operations. 

Table 18. Number of Live and Simulated Flights providing Data for TCL3 by SAA Test 

NASA Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Total 

SAA1 8 8 

SAA2 8 4 12 

SAA3 2 5 7 

SAA4 3 6 9 

SAA5 2 2 4 

SAA6 1 1 

All 

SAA 11 14 16 41 

53 
 

 

 

 

 

 

 

 

6.3 General SAA Feedback 

A standard set of nine questions was asked at the end of every SAA survey, requesting that 

participants rate the information they used, timeliness of actions, their opinions of the information 

they received through the USS clients, and their safety concerns. Sometimes discussion in debriefs 

took a more general focus and was not directed specifically at the test the site had just flown. 

6.3.1 Information Properties 

Participants were asked what information they used to stay within their flight geographies for the 

SAA flight tests. They were given eight items of information to choose from, and participants 

indicated that they used six of these (Figure 21) with two additional items of information that had 

not been listed (see “other” on Figure 21)—visual contact and the “well clear” radius. They reported 

having used vehicle position reports and volumes available on their USS client to stay within their 

flight geographies most often. 

Figure 21. Information used to stay within flight boundary on SAA flights in TCL3 

shown by type of SAA test (n = 14). 

Participants were invited to rate eight properties of the UTM information they received through their 

USS (Figure 22). On average, participants rated all the UTM information they received as “okay” 

with means varying from 3.5 for usefulness of information for planning to 5.1 for accuracy of 

information, on average. Participants gave the highest ratings (on average) after SAA 2 and the 

lowest after SAA 1. 

54 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

solely relying on UTM to provide their safety element during the SAA flight tests. The majority of 

respondents (53%) reported they would be “very concerned” with this situation. From the comments 

respondents made listed in Table 19, they were concerned both about the reliability of their USS 

clients and the reliability of some telemetry data. 

Table 19. Reasons for Not Relying on UTM for Safety (n =13) 

SAA Test Test Site Comment about Relying on UTM for Safety 

1 6 

The false alerts and “bugginess” of the USS makes me not inclined to trust it 

entirely. Most of the needed conflict mitigation constructs are in place or in 

progress but the software has not had time to mature to the needed level of 

quality assurance. 

1 6 

The XXX client lacked the features required for safe operation of the XXX 

aircraft. Additionally alerting was not implemented. There was a loss in 

usability and a lack of value in the XXX client. While it looked good in some 

places, it lacked basic functionality throughout. 

2 1 

Two streams of data relayed the aircraft’s position to the USS client; both XXX 

GNSS position and GNSS position from ADS-B. ADS-B position relayed from 

the local ADS-B receiver showed latency issues when viewed from the client 

portal. If I was relying solely on position of other aircraft from ADS-B alone, I 

would be worried if I did not have a visual on mine or other aircraft. 

2 5 Since we had a MAC in the air, yes absolutely I would feel concerned about the 

flight operation we conducted. 

3 6 

Due to the nature of testing and how the USS reliability was intermittent I 

would not have relied on it. Some connectivity loss experienced was only 

apparent when telemetry stopped updating on the USS client display. 

3 6 

At times it was difficult to determine a legitimate radar track from false tracks. 

This is mostly radar tuning related, not UTM related. Presentation of tracks 

could graphically represent and emphasize track age and confidence (as 

reported by the radar); this would make it easier for the operator to visually 

identify a track belonging to an aircraft vs. a false track. 

5 6 App currently lacks enough functionality for pilot to develop trust in the system 

6 6 XXX client provided no information about other aircraft or warning about 

conflicts 

6.4 Crew Feedback from Debrief Discussions 

In addition to general comments and survey responses from crews on the SAA tests, aspects of the 

debrief discussions will be examined below and organized as feedback on the SAA concepts, 

technologies, alerting, and operator experiences. 

6.4.1 Concept 

Concept: Participation 

Participants discussed the approach of the different types of users who fly UAS. They noted that 

crews who currently comply with Part 107 are “good stewards” of the national airspace. They are the 

group who will file, and then fly, flight plans, and will comply with new regulations on SAA 

56 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

technology requirements. However, they argued that the hobby market, or even the Part 107 pilots 

who fly racing drones, may have a different mentality and take the position that they are having no 

trouble flying without UTM and they should be exempt. Crews felt, that this subgroup may be more 

skeptical and less accepting of having to add equipage to their UAV. 

Other players who will affect how the rules for UTM airspace develop are manned aircraft and the 

regulators. Ideally, UTM airspace would be a fully cooperative national airspace (or at least have 

zones that are designated fully cooperative) integrated with the airspace above, but manned aircraft 

may not want to be cooperative at all times and may resist this, making SAA technology on UAVs 

necessary. 

Concept: Airspace Definition 

Regulations should determine which users can fly where, but crews did not think this is the way the 

UAS airspace will be defined. They see the rules of UAS airspace usage being defined by equipage 

requirements, where the level of access and risks that a mission requires determines the level of 

equipment that needs to be on a vehicle. For example, if you wanted to fly BVLOS, there would be a 

list of equipment requirements that you would have to meet to do this. Given the number of groups 

who will fly UAS, crews agreed that there needs to be a system like UTM in place, and that the 

separation aspect of UTM is the real benefit. 

Concept: Vehicle Capabilities Procedures 

Crews proposed that there must be a hierarchy of which type of vehicle should give way to others. 

As UAV are more easily maneuvered, and manned aircraft are easier to see (UAV are very difficult 

to see from a manned aircraft), the UAV should have to move to avoid the manned aircraft. 

Crews debated the best maneuver for sense and avoid. They cited a case during one of their flight 

tests where the intruding aircraft was between the UAV and its GCS. This PIC thought that the safest 

maneuver would be to fly away at a 90o angle because the other vehicle would not be able to turn to 

intercept him on that course, and therefore this would put the UAV well-clear of the other vehicle. 

Others argued that you cannot predict the flight path of the other/intruder vehicle, and if it was 

taking a curving course the two vehicles could meet even with a 90o maneuver and felt that hovering 

in place would be safer. 

Concept: Crew Procedures 

One site wrote their own SAA procedures to guide their actions in their flight tests. In general, they 

advised their crews to stay in communication with each other and think ahead to try to resolve issues 

before they became problems. One of the procedures, in the case where an incursion has happened, 

stated that when an aircraft or target moves inside a warning area that the PIC defines during 

preflight, then the pilot, who has manual control of their system, has to give way and attempt to 

maneuver out of the warning area. 

Concept: Risk Mitigation 

Participants suggested that work on use cases is an important part of convincing the public to accept 

the risk of UAS flying in the urban environment. They suggested that, initially, the industry can 

mitigate the risks with other means, like limiting locations and flights, until the technology matures 

to where you can fly in the urban canyon. As technology develops, truly autonomous flights can take 

place. They suggested that the general public tends to immediately worry about privacy issues when 

UAS are mentioned and do not yet see the benefits of this technology. 

57 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Concept: Automation Toolset 

Crews suggested that automation should be introduced to reduce their load, and in particular, for the 

pilot. The advantage of automation is that the pilot will be able to focus on the aim of their mission 

and they can hand the details of how to achieve that over to the automation. Ideally UTM would be 

transparent to the user, for example, it would automatically assign your mission a volume with all 

the required parameters, as the pilot completes flight path planning. More load would be relieved if 

UTM automation took over some of the flight decision making. If there is an issue, currently the 

pilot reviews the situation and decides whether there is a safe place to land nearby. This is one area 

where the transparency of UTM and more automation could reduce workload by identifying a safe 

landing area, like NASA Langley’s “safe to ditch.” Crews argued that, as flights go beyond VLOS, 

the system will need to have more of this kind of automation, including automated de-confliction, 

because the pilot will be one step removed and will find it more difficult to provide correction or 

evasive action. 

Some crews said that UAS should need to have sense and avoid capabilities to be allowed to 

participate in BVLOS. They recommended that the SAA technology should be self-contained, 

automated, and require no human intervention, not only because humans are “slow and miss things” 

but also in case the GCS link with the UAV is lost. Crews argued that if you lose linkage, then your 

vehicle becomes rogue and, if it is a busy airspace, that can have serious consequences. So, the UAV 

has to be able to avoid another aircraft automatically. Others in the group disagreed, saying that the 

human-in-the-loop is important in the case of a lost link, but participants countered that there is very 

little chance of a lost link when crews have thoroughly planned their flight. Then the hazard is 

mainly someone else jamming the signals. 

They reasoned that there is already an industry-driven standard autoflight software, a flight planning 

tool and ground control station tool, but what they lack is cloud-based C2 and multi-vehicle C2. 

These are the properties that UTM is bringing to the table. Participants pointed out that, as flight 

crews, they are “stuck in the middle,” being invested in commercial off the shelf software (COTS) 

GCS tools and needing the feature set of their USS, which means they have to drive both softwares 

at the same time, and this is very challenging. 

Crews said that it is good to have as much situation awareness as possible and that every new tool 

that can enhance your SA is a good tool. Having better SA is necessary, especially as the 

environment gets more complex. They liked the idea of being aware of the location of other traffic, 

noting that in manned aviation, it is possible to know where other aircraft are through ADS-B. Crews 

stressed that having situation awareness is critical when you are flying BVLOS particularly as UAS 

can fly at high speeds and can rapidly enter the area near your path. 

6.4.2 Technology 

Equipment: DSRC 

In one discussion the crew said they did not think that DSRC added to UTM, as the user can already 

get telemetry data from other sources. Although DSRC may not add anything by being another 

source of position data, its advantage would be that the UTM community could leverage existing 

infrastructure put in place by other entities (e.g., Department of Transportation). 

Crews thought the advantage of DSRC was the traffic and hazard awareness that it provided. They 

liked the idea of knowing the location of other UAS that may interfere with their mission. Crews 

noted that because it is their responsibility to see and avoid other traffic, they will always need to be 

58 
 

 

 

 

 

 

 

 

 

 

 

 

on their guard and looking for UAVs that are not being flown cooperatively, that they will always 

have to fly as though they will have to see and avoid other traffic even if everyone is flying 

cooperatively. Technologies like DSRC that enable crews to see and avoid are important when 

flying BVLOS and will enable the pilot to choose a conflict resolution should they need to. 

However, one participant noted that the full value of DSRC was not realized in their flight tests 

because the sensor data was being sent to the server first, processed through the collision avoidance 

software, and then back out to the USS client. They said that their GCS should also receive the raw 

DSRC data from their vehicle because this would provide faster alerting. Crews noted that the DSRC 

tracks were very clear and they could have used this information to make manual deconfliction 

decisions. 

Equipment: ADS-B 

Participants said that UAV are difficult to see when they are in flight. ADS-B was invaluable for 

flight identification, and crews used their ADS-B-fed displays heavily to see where target aircraft 

were. Even when they knew the general direction to look, participants said it was difficult to see the 

vehicle. Crews discussed one SAA flight where the manned aircraft flew straight at the UAS. Crews 

were surprised because they had not anticipated that the manned aircraft would be able to pinpoint 

where the UAS was just from its ADS-B signal. The direct line flight meant that the UAS-PIC was 

unable to execute the RTB avoidance maneuver that had been planned, and so made the secondary 

maneuver of descending. 

It took one test site a little while to calibrate their ADS-B because the timestamp on the reports was 

incorrect, and initially one UAV looked like multiple vehicles. They filtered the incoming telemetry 

differently to overcome this. Another crew thought they had an intruder very close to them but it was 

resolved because the GCSO had not tied the vehicle’s ADS-B readout to its GUFI in the USS client 

and so the client read the ADS-B as indicating a second, intruding, vehicle. They resolved the 

problem quickly because the ADS-B was tied into the UAV’s tail number and they realized the issue 

as the client announced the “intruder”. 

Equipment: Radar 

With respect to the purpose of the test, crews were happy to report that the ground-radar tracks they 

saw were close to their GPS-driven UAV tracks, and they confirmed this for multiple vehicles. 

However, the only information they received were the “blips,” without any additional information. 

They were concerned because UTM/NASA did not see these tracks for all of the UAS, which meant 

that the USS client system was not reporting on all of the radar data it received. 

Crews noted that radar inherently has false alarms for targets that are real, but not relevant, as radar 

tracks are shown for all airborne vehicles and entities (e.g., birds). While most of the tracks may not 

be potential conflicting targets, crews suggested that it is useful information to see everything 

airborne because it provides additional information about which altitudes have more traffic, and 

about areas the PIC might want to avoid because they are busier. 

Automation: Sensor Trust Reliability 

Crews noted that the reliability and availability of sensor data was key because for SAA you need to 

detect other airborne objects at suitable ranges in order to have sufficient time to maneuver. Being 

sure that the sensor data are reliable allows the user to have confidence in threat detection. Crews 

pointed out that pilots and GCSOs want to receive telemetry data and want to be able to trust it, but 

59 
 

 

 

 

 

 

 

 

 

 

that they did not care what sensor it came from and did not need to know the statistics per sample. 

They just wanted a general impression of the overall reliability of the data. That said, crews tried to 

compare DSRC to radar feeds and found it difficult to compare sensors because an active sensor, 

like DSRC, requires other UAS/aircraft operators to be using it (buy in) for others to receive any 

surveillance. Radar is passive, in the sense that it does not require on-vehicle equipage, which means 

you can receive surveillance of others even if they have low equipage or are not participating. One 

participant suggested that a graphical depiction of good/bad would be sufficient for this, and that it 

would be helpful if the good/bad rating was calibrated against an industry standard that characterizes 

the reliability of sensors. Another participant suggested that each (radar) track could be color coded 

to reflect the confidence-level of the data that had generated it. 

Automation: “Safe Zone” Tools 

The various test sites developed different approaches to depict “safe zones.” For example, one site 

employed a spherical protected/ alerting zone for SAA on the USS client, defining a radius to 

generate the protected zone, which was applied in all directions around the UAV, and changed 

colors (from blue to red) as an intruder flew into this area. However, users commented that it was 

difficult to develop good situation awareness from a 2D display of the sphere, as they only saw the 

lateral protected zone out from the waist of the vehicle drawn on the screen. It was difficult to 

reconcile or calculate geometrical distances mentally while the UAV was in flight, as when a vehicle 

was just above or just below the ownship. It appeared that the intruder had breached the protected 

zone but actually was still outside the ownship alerting boundary, because the sphere was laterally 

narrower at that point. A PIC noted that this was unnerving, because they did not have a good 

altitude indication on the USS client, making it very difficult to judge whether vehicles were actually 

in conflict or not. 

Using the working assumption discussed above—that SAA functions had to be either wholly or 

partly automated—another test site developed an approach for the way such technology would work 

and, although they could not build it fully, they emulated an automated avoidance function. Their 

concept for automated aircraft avoidance employed a display showing a cylindrical protected area, 

where one could set certain parameters of the cylinder to avoid other aircraft. They added SAA 

features to their UAS software—where the virtual cylinder could, potentially, trigger automatic 

avoidance maneuvers to avoid incoming conflicting aircraft that fly through the boundary. The 

current software development allowed the user to set a radius and altitude to the cylinder. The 

cylinder was color coded and when it turned red, the PIC knew that they had to start avoidance 

maneuvers. In addition, the intruding aircraft icons on the display went from green to amber to red as 

they traveled through the warning zone to the protected cylinder of the ownship. The test site tried to 

simulate automated avoidance maneuvers11 by having the PIC do what they would have 

programmed the UAV to do, for example, a command to automatically go to a predetermined point 

on the map at a given altitude. There was also a 30 second buffer—that the UAV would execute a 

maneuver if the conflicting vehicle was within 30 seconds. 

6.4.3 Operator Experience 

Operator Experience: Situation Awareness 

Participants talked about how they allocated their attention while flying. One participant said he was 

trying to maintain the best situation awareness he could, and the cost of that was that he was paying 

11 The avoid action is where the avoidance actually occurs after the intruder has breached the 

warning zone or cylinder. 

60 
 

 

 

 

 

 

 

 

 

 

less attention to his tablet and tool interfaces and more attention to the sky and his remote-control 

pad (operations were VLOS). He wanted to keep an awareness of the other UAV while maintaining 

a visual on his own UAV. He said if something went wrong, being able to see the vehicle directly 

would be faster than the other pilot calling him on the radio. 

Having tested out their conflict alerting software, crews noted that their UAS software did not 

provide as much vehicle information and sense and avoid parameters as they would like to know. 

Crews listed a number of USS features and functions that they thought would improve their situation 

awareness for conflict situations. They thought altitude, speed, heading, vehicle position on the map, 

velocity and their vehicle’s protected zone would be useful. For the intruding aircraft (manned or 

UAS), they thought the target’s position, altitude and having a longer length on the intruder’s history 

tail would be useful, as would velocity vector, which they could calculate relative to a warning area. 

Ideally, the display would be a predictive vector to give some idea of the vehicle’s forward flight 

path, but a history also provides good information. The crews stressed the need for the history tail to 

be long, so that if you have looked away from the display, there is still enough information on the 

screen when you come back to it to make predictions. One participant discussed that the pilot needs 

to know the difference between his/her ownship and the intruder regardless of whether the data is 

from airborne or ground sensors. This distinction was clearer through ground-based radar than 

through the airborne sensor displays (DSRC) they used. 

Crews also stated the need for audio/ audible alerting because they could not watch one display the 

whole time (see earlier conversation) and having the tool annunciate conflict alerts would have 

assisted with pilot workload. During their flight tests, none of the participants heard audio-alerts for 

the conflicts they were flying, although they thought they had heard alerts in previous shakedowns 

and testing. 

Operator Experience: Workload 

A real challenge for pilots is that they are managing two sets of software while they are flying their 

vehicle—UTM software and their flight automation software. The chief test site coordinator noted 

that this created an unacceptable level of PIC workload. The UTM software supplies situation 

awareness, and not flight controls, but pilots are unable to capitalize on this because they are 

occupied flying their vehicles and looking at a different display. 

Crews noted that, as pilot workload is an important thing to manage, having a very good alerting and 

a visual display is critical. Comparing the airborne (DSRC) and ground based (radar) surveillance 

feeds, one participant thought that there might be differences in pilot workload depending on the 

data source. For example, the pilot needs to know the difference between his own-ship and any 

intruder. As the ground-based radar reports intruder vehicle position relative to the radar rather than 

relative to the ownship, that might increase pilot workload because PIC will need to do a calculation 

to determine the severity of the conflict. 

Operator Experience: Trust 

Participants noted that using UAS in urban environments depends on users’ trust in autonomous 

systems and the extent to which they have been proven to mitigate the set of potential contingencies 

that could be required. 

Crews complained that their USS did not provide them with enough information about the status of 

their volume. On one client, status was shown through a text note on the side of the page saying 

61 
 

 

 

 

 

 

 

 

 

 

 

 

“accepted” or “activated” or “rogue,” which crews said was not salient enough (note that this 

comment is for one particular client not for clients in general). On their other client, the status was 

noted on a separate window from the one used for flying and did not seem to change during flight. 

One piece of information crews would have liked, in particular, was whether the rogue state was due 

to vertical or lateral (or presumably, temporal) excursions. Another was to be given an indication of 

the direction the UAV needs to move to get back inside its volume. They noted that all their rogue 

statuses were unintentional drifts of the vehicle outside its volume boundary either because they 

were flying too close to the volume bounds or because the takeoff/ landing was difficult to judge. 

Crews noted that they had also seen many false rogue states during their shakedowns and testing 

(where they could verify that their UAV was not rogue), to the point where they tended to ignore 

rogue status when it alerted on their USS client. The team was investigating why so many rogue 

states happened with the USS client developer but had not resolved the issue for the flight test. The 

chief test site coordinator noted that effort needs to be made to build pilots’ confidence in UTM 

capabilities. 

Crews were concerned about accuracy of altitude measurement on the UAV, as discussed in reports 

for previous flight tests. They were concerned that if one UAV was using geometric altitude and 

another was using ADS-B pressure altitude that the vehicles could potentially climb into each other. 

Consequently, some avoidance procedures did not make sense under certain conditions when the 

altitude measurement method was unknown. Some argued that, to address this, the FAA will need to 

require one type of altitude measurement on UAS (probably barometric pressure altitude). Others 

said that avoidance procedures, like those that apply to manned aircraft, will need to be put in place. 

One pilot advocated that “stop and wait” should always be the first action that a UAS takes. 

6.4.4 Automation 

Automation: Issues using USS Clients 

At one site, one of the vehicles was flown using a different client than the others. This client only 

had outbound data, which meant that this PIC did not receive any alerts or notifications. This PIC 

noted that it was frustrating during operations when other crews were seeing alerts and he had no 

indications. He worked with the developers to add a few features to his client, e.g., to show a lost 

connection with the server but, he said, that it was still not enough to give him awareness about 

when the alerting was operating. 

Other crews reported having had some confusion regarding functions that seemed to both require 

manual activation and were automatic at some level. Pilots were confused that they had to manually 

activate their volumes but at the same time these were “auto-de-activating” (these volumes 

automatically expired). These issues arose when the flight was delayed and the crews did not realize 

they needed to resubmit their volumes with updated time parameters. Crews would have liked to be 

able to put a delay on the activation of the currently accepted volume, so that the volume did not 

expire and was available to be activated later. 

Users from Test Site 6 commented about the functionality of their USS client. They had some issues 

with clutter when warning boxes popped up in sequence, scrolling down one side of the screen. 

Another issue they found disappointing was that alerts were meant to be annunciated aurally, but 

were not reliable. Some of the participants had heard the alerts when preparing the scenarios but 

during the test the audio did not work. They felt the visual alerting on the USS was good but would 

have been complemented by aural alerting. 

62 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Participants mentioned that knowing where moving cars were on the ground would be useful but that 

improvements in the USS client/UTM are more important for the usability of the system. They noted 

that the UTM system needs to improve in the following areas over the system they used in the flight 

tests: the USS client needs to work properly, it needs additional functionality, and the overhead to 

use it needs to be lower, which could be achieved if the interface was more transparent. 

Users noted that they would like to see history trails or breadcrumbs (vectors) for other aircraft, that 

the pilot wants to make sure they avoid. They would prefer to be able to see other vehicles and plan 

their flight path to avoid them, rather than having to deal with collision avoidance (i.e., that tools are 

good at the sense-and-alert portion of sense-and-avoid). In general, users said that having to input 

data was undesirable and, as much as possible, UTM should work without users having to tend to it, 

providing information with less input from them. For example, the display currently requires that a 

mouse hover over the target to indicate altitudes, it would be nice if this information was 

automatically provided. 

Automation: Tool Trust Reliability 

One participant suggested they needed a predictive capability for SAA, given the current state 

information of aircraft in proximity to each other. He said it would be helpful if the predicted 

trajectory of both aircraft could be displayed and then updated in real time as the pilot is 

maneuvering. This would give the PIC a view of the closure rates and the distance to the closest 

approach. 

During one series of flights, a crew noticed inconsistencies in the acknowledgement of the guided 

mode command when their UAS was flying in auto mode, which they thought was an autoflight 

software issue. However, the test site coordination team thought that it could have been too many 

people crowding around the display in the test site control location to see the radar and observe the 

test. They thought the people occluded the antenna and therefore the commands being sent from the 

station, i.e., the “go to guided” command. 

Automation: Areas for Improvement 

Crews pointed out the need for sense and avoid technology to be small and light is a limiting factor 

currently, as is the onboard computing power. They also noted the lack of readily accessible 

computer vision technology. 

Crews frequently saw telemetry hiccups and small delays, where for a second or two no packets 

were received but after a couple of seconds the telemetry came through again. They discussed that 

these hiccups are acceptable with UAV that are traveling more slowly (e.g., at 6m/s) but are 

unacceptable for vehicles traveling faster (e.g., at 50m/s) because the closure rate is so quick that 

you need to be able to see the path of the vehicle at all times. They stated that for these situations, 

the protected cylinder would have to be very large to accommodate the closure rate of fast-moving 

vehicles if the PIC was manually completing SAA, likely so large that this approach may not work. 

Again, they noted that automated SAA may be able to overcome this issue. 

As an alternative approach, one participant noted that having a leader line on every vehicle, that 

predicts its trajectory, would be very helpful when vehicles are moving quickly. They argued that a 

leader line may be more useful than a protected cylinder for SAA. 

63 
 

 

 

 

 

 

 

 

 

 

 

 

 

The flight tests revealed a number of small issues with the UAS and USS client software. Crews 

found that, initially, they could not see all the aircraft on their GCS automation displays because they 

had loaded a version of the software that was too recent. They also lost their GPS telemetry for a 

little while but were confused because they were locked on to enough satellites for the GPS to be 

coming through. One PIC thought that someone local to them was jamming the GPS signals. 

As in all the tests where crews were exercising their software, one of the SAA test sites found a 

number of functions that did not work as expected and possible bugs in their system. One general 

issue with the automated logic was found when the test site was testing how the vehicle moved 

perpendicularly in its volume. On another flight, the GCS lost the telemetry link on their UAV, 

saying it “strangely disconnected,” forcing them to land with a loss of communications. 

Participants brought up the additional workload of having to monitor both a USS client and their 

auto-flight software. They lamented that the larger UAV manufacturers had not integrated UTM 

functionality into their auto-flight packages, and that working with UTM would become much easier 

if they only had one display to monitor12. If manufacturers were building UTM features into their 

products, then UTM integration would be more seamless. For now, USS developers are having to 

build USS components as tools separate from the autoflight software. Pilots felt they would utilize 

the USS more if it was integrated with their auto-flight software. They suggested integrating 

software the other way around—pulling auto-flight software into the USS client (but noted, that 

where this has been done currently the functionality on the USS is not as good as on the auto-flight 

software). They argued that, since integration of functions is unlikely, UTM needs to work on 

making its (UTM) functionality transparent to the user—essentially reducing the overhead of using 

UTM as much as possible. 

Crews debated the utility of cloud-based servers. The benefits are the functionality that can be 

gained with a cloud-based system that allows users to revoke remote control, for example, for 

unintended UAS placement. The negative aspects of cloud-based servers arise if the server develops 

a fault, then all operations are affected. Crews advocated for an assessment to calculate whether the 

benefits of cloud-based systems outweigh the risk and, if this is situation-specific, the conditions 

under which you would take on that risk. 

6.4.5 Alerting 

Users of the spherical alerting zone noted that their alerting was “basic” (e.g., only being able to 

define one radius), and suggested having more than one alerting zone—a warning and then a 

protected region—with a sequence of alerts, or a variable sphere size based on the situation, which 

would make the system more useful. They noted a “hockey puck” (cylinder) shape for the protected 

area would be easier to use than a sphere, and that the saliency of the alerts and their ability to get 

pilots’ attention in an appropriate way needs improvement. They also noted that integrating radar 

and DSRC feeds with the alerting logic would be nice but would be difficult to achieve. 

The protected cylinder that Test Site 1 built for their SAA maneuvers had a height of 100m above 

and below the vehicle. However, crews felt that when altitudes are being measured in a number of 

different ways (geometric, pressure, etc.), that a 100m protected zone does not guarantee vehicle 

12 One participant noted that two of the major ground control station developers/ manufacturers are 

not developing UTM functionality. 

64 

Mean SA rating 

Very 7 

high 

6 

5 

4 

3 

2 

Very 1 low 

 

 

 

 

 

 

 

 

 

 

 

awareness of conflict clarity of conflict alert effectiveness of alert 

Situation awareness of aspects of the conflict 

Figure 24. Awareness of conflict situations during SAA 1 flight tests (Note: n = 2-4). 

One of the two respondents said s/he could see the ground vehicle’s DSRC messages through the 

USS client, which showed him/her “a little information.” The other participant could not see these 

messages, as their USS client display did not show any information about the ground vehicle. This 

meant that the two participants gave different answers to the questions that followed about how 

clearly overflights were displayed and how effectively they were alerted by the client to trajectories 

over the ground vehicle. 

6.6 SAA 5 

The interoperability between all the mitigations discussed above was the focus of SAA 5, with test 

sites looking at USS to USS interaction. Survey questions asked about situation awareness, the 

timeliness and clarity of messages and the resolution maneuvers made. Only two responses were 

received on the survey for this flight test. 

One of the two respondents to this set of survey questions said they were informed about intruding 

vehicles by their USS Operator, and the other did not receive any USS information. The participants 

were not aware of the USS message stream, as neither of them were USS Operators, however, they 

reported they were “somewhat aware” of their vehicle. Because they could not see a USS client, 

participants reported rogue states were “not at all” clear to them. They did have a more favorable 

view of their operations, estimating that their replanning was “reasonably effective” and that their 

resolutions to airspace intrusions were “very effective.” 

6.7 SAA Test Summary 

SAA testing was exploratory, with test sites using new equipment (to them) to sense and avoid a 

variety of vehicles. This experience not only highlighted features and issues with SAA tools but also 

helped crews to reflect on how they are using their GCS, USS clients and the UTM concept. Some of 

the major points included: 

• The telemetry that provides additional situation awareness can come from a range of 

66 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

sensors, one type of data was not clearly better than another, when comparing across tests. 

But, the data does need to be reliable and consistent. 

– Reliability: Some SAA data had uncertain reliability, making those crews unsure 

about all of the data they received. Implication: As above, this suggests continued 

progress to improve the reliability of SAA data. 

– Trust: Users did not always trust the telemetry and sensor data enough to rely on it. 

Implication: The accuracy of information provided through UTM would benefit 

from improvement to make it as consistent as possible. 

• Test sites liked the additional sensor tools they had that provided increased situation 

awareness about the airspace they were flying in. 

– SA: Some crews were not confident enough to fly SAA tests while BVLOS, 

suggesting they did not trust their tools enough to move away from visual contact. 

Implication: Continued progress is needed to improve the reliability of SAA data. 

– Trust: As BVLOS is a higher risk environment than VLOS, this is the type of flight 

where SAA will be needed most and can be most informative. Suggestion: 

Continue work to improve both training and the reliability of SAA data, which may 

counteract the lack of trust experienced by some of the crews in this study. 

• Test sites liked the idea of SAA tools to alert them when intruders were approaching their 

vehicle too closely but reported a number of issues with this new-to-UTM technology, 

ranging from telemetry reliability issues to interface interaction issues. They concluded, in 

general, that these tools need more development to become truly useful. 

– Usability: Some USS displays either lacked SAA functions and/or were cluttered. 

Implications are that it could be advantageous to continue to improve USS GUIs. 

– Functionality: SAA technologies were all good for awareness, but some users were 

just getting to know how to use them and which were best for what situation. 

Implications are that continued testing of SAA technologies could be beneficial. 

– Workload: Crews commented on the level of workload required to monitor all the 

displays in a GCS. Moving forward, in addition to addressing the items above, 

consolidating data onto fewer displays could alleviate workload. 

• An area where crews were uncertain was how to react to SAA events. Discussion 

highlighted the variety of approaches crews had considered that could be used to respond to 

conflict situations. 

– Procedures: Crews created their own procedures for avoiding conflict and giving 

way. Moving forward, a consistent set of procedures should be tested and evaluated. 

– Public and Hobby Users: From their experience of flying SAA, crews noted that 

having operators outside the UTM system will be problematic for SAA. Suggestion: 

Offer hobbyists incentives to encourage full participation in UTM. 

7. Communication Navigation Surveillance Tests 

Three CNS tests were conducted to examine technologies for communication and navigation by 

measuring data elements, such as the accuracy, latency, and switchover effectiveness. For the first 

test, CNS 1, the test sites attempted to maintain positive control of the vehicle during a scripted 

primary C2 link failure, and subsequently switchover to a backup link. To assess whether the 

redundant C2 was successful, a maneuver command was sent to the vehicle, and the vehicle’s 

67 
 

 

 

 

 

 

 

 

 

 

 

 

 

response was confirmed. The ability to remain within a flight geography using GNSS navigation was 

the focus of the CNS 2 test. The test sites were encouraged to utilize manmade structures to naturally 

inhibit or block signals between the vehicle and GPS satellites, or create a multipath effect. For CNS 

3, vehicles were equipped with a radio frequency (RF) sensor payload, and gathered data at three 

different altitudes to characterize the RF environment and inform the impact on C2 communication 

in low altitude. 

Five of the six test sites were awarded two or more CNS tests. In total, 12 CNS tests were awarded, 

and these awards were distributed so that each CNS test was performed by three to five test sites. 

Test sites conducted these 12 sets of tests over 18 calendar days, across two months. For some of the 

tests, an AOL observer was present in the field and took notes. For all the tests, participants 

completed a survey at the end of the day. Also, at the end of a number of flight days, crews took part 

in a debrief, either with the onsite AOL representative or over the telephone. These discussions and 

responses are reported below. 

7.1 Method 

7.1.1 Surveys 

There were 30 questions in the CNS survey, with the questions preselected to show only between 10 

and 25 of these questions depending on the test. All CNS tests received 10 general questions, and 

only CNS1 and CNS2 received additional test specific questions, 15 and 20, respectively. Many 

items asked the participant to rate their answers on a 1–7 scale, where 7 was high, or positive, and 1 

was low, or negative. Other question types were multiple choice and free-response. Ninety surveys 

were started in total across the five sites. All 90 surveys had some data entered but not all were 

complete. The greatest number of surveys started at one site was 25 and the fewest was 11. 

7.1.2 Debriefs 

There were nine debriefs that focused on the CNS tests that were hosted across the five test sites. 

Eight were specific debriefs focused on one CNS test while one touched on all three CNS tests (see 

Appendix 15). Numbers of discussion prompts varied, as researchers wanted crews to discuss and 

explore the topics within the time available. Prompts focused on operator situation awareness, the 

UTM concept, usefulness of sensor data, and technical issues of concern. 

7.2 CNS Metadata 

There were 134 operations submitted to UTM by the test sites to support CNS testing. Sixty-four 

operations were submitted for CNS 1, 14 for CNS 2, and 55 for CNS 3. See Table 20 for a further 

breakdown of submitted operations. 

Table 20. Number of Live and Simulated Flights Providing Data for TCL3 by CNS Test 

NASA Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Total 

CNS 1 21 8 19 15 1 64 

CNS 2 6 4 4 14 

CNS 3 3 21 25 6 55 

All CNS 27 15 40 40 11 133 

68 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

8 

Very 7 

CNS 1 CNS 2 CNS 3 

high 

6 

5 rating Mean 

4 

3 

2 

Very 1 low Awareness of UTM states Workload at busiest time 

Operator experiences 

Figure 27. Operator workload and UTM awareness during CNS testing in TCL3 

shown by type of CNS test (n = 49–75). Note: Rating scale was 1–7, yaxis is longer to show SD. 

7.3.3 Technology: RF Spectrum 

Participants debated which portion of the RF spectrum UAS are going to use in the future. Crews 

predicted that in the future, when there are many UAS in the airspace, then the RF bandwidth will be 

over-taxed and the command for one aircraft will possibly overlap onto another aircraft, which could 

lead to un-commanded movement. They noted that encryption will help but also that crews may 

need to declare the frequency they are using, which most will not want to do for privacy and security 

reasons. 

Currently UAS use industry, scientific and medical (ISM) bands but they predicted that with the 

number of UAS flying these bands will become overloaded. One participant noted that this situation 

will improve when an aviation grade spectrum is available for UAS, but the team speculated there 

will still be a question about whether long term evolution (LTE) or the ISM band might still be used 

by smaller UAVs. 

7.4 CNS 1: Effectiveness of Redundant C2 in Maintaining Operational Control of UAV 

The CNS 1 test was designed to evaluate the effectiveness of redundant C2 tools in maintaining 

operational control of UAS. Crews were required to use two C2 tools on their vehicle (e.g., two 

point-to-point radios or a radio and a cell network), send a UAV maneuver command using their 

primary link to verify execution of the sent command via vehicle telemetry, and then to turn this link 

off (simulating a failure of the primary link) and re-execute the maneuver, again collecting telemetry 

data. In some cases, the test sites performed a natural link transition, where they would fly to a point 

where the primary signal was degraded enough to trigger an automatic switchover. It should be 

noted that the impact of a real failure varied across different UAS. For those whose systems were 

redundant, CNS 1 was a test of the auto-switch capability on their UAS to just use one radio, rather 

than two. This should not result in an interruption in telemetry, and was not an event that represented 

a failure for the crew. A failure would have been for the second radio to fail as well. For those crews 

who were using a single-link system, CNS 1 was a C2 failure, in the sense of an interruption in 

71 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

telemetry, with a switch to the back-up radio/network restoring the telemetry. This functional 

difference gave crews different perspectives and led to them responding differently to survey and 

debrief questions. 

Survey questions asked about the process of switching C2 links, crew procedures and users’ 

situation awareness. In debriefs, crews discussed alerting, GCS functionality as a model for USS 

clients, and C2 awareness (Table 21). 

Table 21. CNS 1: Discussion Topics and their Notional Categorization into 

Information Themes 

Debrief Topic Category Site 1 Site 2 Site 3 Site 4 Site 5 

Information property 7 11 

Information meets user 6 1 9 

Operator status 1 11 

Methods 1 2 2 

Automation 8 6 6 

Concept 5 3 

From the survey, 86% of the respondents reported that the UAV they were crewing automatically 

switched to its back up link (see Appendix 14 for details). Six respondents indicated that the C2 

failure was shown on their GCS auto-flight software, through a message or icon, five reported that 

nothing was flagged, and a sixth had a manual system. One person noted that their USS client 

showed an alert message, while another reported that their USS client did not receive C2 linkmessages. Forty seven percent of respondents indicated that their system automatically executed a 

contingency flight plan when it lost its primary C2 link, and 72% of respondents also reported that 

their UAS automatically continued on its original flight plan after switching to a back-up C2 link. 

7.4.1 Methods: Management of Failure 

Participants were asked a number of questions in the survey to establish how they were managing 

the C2 link failure. Sixty-six percent of survey respondents said they were monitoring the health of 

their ground-stations’ C2 links with their UAVs. However, only 9% used the USS client to do this 

(confirming their USS client showed the C2 link failure), everyone else reported using their GCS 

auto-flight software for this. The 33% who were not monitoring the C2 link reported that it was not 

part of their role to do this, with one PIC explaining that he was not monitoring the C2 link because 

he was relying on his GCSO to tell him if they were switching between links. When the C2 link was 

failed, only 9% of the respondents did not know it had failed, and 27% reported they knew of the 

failure because they were told by a teammate. Surprisingly, of the 53% of respondents (n = 21) who 

said they became aware of the failure through their tools, 80% of this group (n = 17) reported they 

were made aware of the failure via their USS client specifically. (This is only surprising because 

relatively few people reported they were monitoring the USS client for system health and reported 

failure messages coming through on their GCS, see above and discussion below.) They explained 

72 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

example, at one test site, there was a visual indicator on their GCS toolkit that simply turned red 

when there was a C2 failure of that vehicle. Another crewmember suggested a lower time limit for 

reporting lost link, and that one second telemetry drops should not be reported (see below that 

systems follow this suggestion to some degree). 

For other UAS, another PIC emphasized that he would want an extremely low false alarm rate if 

every lost link was going to be broadcast as an alert. Crews pointed out that it is not worth 

broadcasting information that does not affect anyone else, and this would avoid unnecessary 

distractions. Participants argued that, if you still have one C2 link, are in control of your vehicle, and 

are inside your volume, other operators do not need to know you lost your primary radio. Crews said 

they only needed to know about others’ lost link if that operation is not being tracked in any other 

way, or if it would affect their airspace, or the UAV was having trouble (presumably loss of control). 

In the case where the other UAV with the lost link would affect local operations, crews said they 

would want to know as much as possible about that flight. For example, it would be important to 

know if a lost-link-UAV is executing a total loss of link procedure, as this state would make the 

aircraft position uncertain and it would be unable to react to commands. One site said that lost link 

situations may be best shared USS-to-USS, rather than directly to the UAS operator, and then have 

the information filtered down to the appropriate users. They specified that might it be helpful to 

define a level of priority or criticality to avoid clutter arising from C2 link issues of other operations. 

7.4.4 Information: SDSP 

One suggestion for our SDSP was a strength-of-signal heat map showing the spot where radio 

coverage is good and poor within a wider area. Crews noted that a large operator or dispatch 

position, deploying many UAS, might find information like this very useful. Participants also said 

that service providers of the C2 link, as a service that UAS operators will subscribe to, should have a 

way of showing their users when a link is down or impaired. 

7.4.5 Automation: GCS 

GCSs contained the primary tools through which crews monitored their lost link status. Crews 

described both the representation of lost link status and when this was triggered on their GCS. Many 

crews have a redundant C2 link, where all telemetry sources are connected and the crew is able to 

see the health of each link, sometimes as the number of satellites acquired or as a general rating of 

overall signal strength, depending on the system. Some systems have an auto-switch capability 

where the UAS switches itself to use the best signal available. Systems also vary in their time 

threshold for a lost link alert, e.g., 3 or 10 seconds. If the link is lost for less than 10 seconds (or 3), 

the system does not show an alert to the user. The PIC in the debrief agreed with a lower limit to 

alerting, saying they definitely do not need to be told about lost links less than a few seconds long. 

However, in a situation where a crew needed to reboot a system, which would take a relatively long 

time (e.g., 45 sec), this should be alerted to other operations that are in close proximity. All these 

features reduce the number of alerts presented to the user and led to the crew reports, noted above, 

that they were not aware of the primary radio failure. For those whose systems did not automatically 

switch, crews reported knowing they had successfully switched to their back up radio when the 

telemetry data on their GCS re-started. 

Crews described that the strength of their C2 link was shown as a colored bar on some GCS. A good 

connection was represented in green and it turned a series of colors and then to blinking red as the 

signal degraded. Others reported that their systems had signal-strength icons, that changed color with 

the link strength, and pop-up windows to alert its status. Crews suggested these features should be 

75 
 

 

 

 

 

 

 

 

 

copied in USS clients, then lost link alerts would display through a USS client as a color change in 

the aircraft icon or a message. 

7.4.6 Automation: USS Client 

Crews from more than one test site reported that their USS client did not provide them with enough/ 

timely information to be useful in CNS 1 testing. Crews reported a large lag when they were using 

the USS for their UAS to switch from the primary C2 link to their back-up, noting that when they 

conducted flight operations through their GCS alone, without the USS involved, the switch-tobackup was a matter of seconds. With the USS in the loop, it took much longer. Crews speculated 

that the amount of telemetry data flowing through the UTM system when many UAV had two radios 

on board and reporting out was enough to slow the USS (UTM) down. 

Two additional general issues with their USS were also reported by some crews. At Test Site 2, 

crews noted that their USS client showed them a view that was too high-level (zoomed out) to be 

useful. The UAV icon was larger than small volumes and they could not see its movement within the 

space, so they consulted their GCS for all the information they wanted. At Test Site 4, the issue was 

different. This site had multiple USSs covering in the same area and those USSs did not share 

information, so crews using one USS did not know when UAV on another USS network had 

experienced lost link. USS Ops reported feeling isolated and listening to the radio to try to glean 

information. 

7.4.7 Automation: USS Issues 

Crews reported a number of issues with their USS that were revealed during their CNS 1 testing. 

One was a NASA issue: that the NASA side of the UTM system had too many teams wanting to 

make data connections and could not support all of them. This resulted in crew submissions being 

rejected but not for a reason they could address. This issue was resolved during TCL3 testing. A 

second issue was the UTM limitation that it could not accept more than ten volumes (see footnote 5 

above). 

7.4.8 Concept: Privacy 

There are two threads to the C2 lost link test. The first is how much crews wanted to know about 

their own UAS, and the second is how much they need to tell other crews. Keeping others informed 

of your C2 link status potentially entails revealing a good amount of data about your operation, and 

many crews were uncomfortable with this. Participants debated about the amount of privacy UAS 

operations should be afforded. One PIC thought it would be okay to show the position of the aircraft 

but not operator details, and suggested using randomized callsigns to protect operator identity. When 

asked specifically about whether they should be required to report a loss of C2 link to the 

community, the group was divided, saying that it depends on the specific situation (as discussed 

above). If you have a secondary link or seamless back-up mode, so there is little to no effect on your 

performance, PIC asked “why would I share that?” But, they agreed that if the failure impaired your 

vehicle’s performance, then you should broadcast this (and will have to follow a public network’s 

rules). Others thought that, regardless of the situation, you would want to broadcast a reduction in 

capability. One participant suggested that UAS could report, along with their position information, 

whether their system is operating in dual-link mode or single-link mode. If the dual-link switched to 

a single-link mode, then other users would be able to infer that the vehicle had experienced a lost 

link. Other participants supported an idea like this because it would provide information but not 

create alerts. (See also discussion above about alerting.) 

76 
 

 

 

 

 

 

 

 

 

 

 

7.4.9 Concept: Participants 

PIC asked how manned aircraft are going to be informed of UAS that are having difficulties, e.g., 

lost link. They argued that helicopter traffic, in particular, either need a way to monitor UAS traffic 

or to be told about unexpected behavior from UAS traffic. 

An issue arose when a site had multiple USS covering the same area. Crews experienced their flight 

volume closing without warning midflight. They had no indication on their USS client that this was 

going to happen or why. It became apparent that crews using another USS covering the same area 

were able to close the volume on the first USS, which enabled them to submit, and have accepted, a 

volume for their vehicles to fly in that same area. These “second USS” crews were unrepentant 

because, in the time it took the teams from the first USS to troubleshoot the issue and then resubmit 

volumes, the second USS crews had completed all their test flights. 

7.4.10 UTM System 

Participants varied in their opinions of the usefulness of UTM. Two opposing perspectives were an 

argument that all UAS operators should have to participate in UTM because having an operator who 

can decide to not be part of the system and then fly in contravention to all the rules of the UTM 

system will defeat its purpose and make the airspace unsafe. The opposing view was that there is no 

benefit to UTM other than a “no guarantees” assurance that the airspace is clear for you. 

Participants want the UTM submission procedure to be more user-friendly. One particular case they 

discussed was moving the window for time of flight, noting that, if they flew before the window 

opened, telemetry was not sent through the system. For example, in the case where the crew needed 

to take off early, due to weather or other constraints, they needed to either make extensive edits in 

UTM or create a whole new operation. 

7.5 CNS 2: Remaining within Flight Geography using GNSS Navigation 

The CNS 2 tests explored the impact of GNSS navigation errors on the ability of UAS to stay within 

their UTM flight geography. To achieve this lack of coverage, one site flew their vehicle in close 

proximity to a structure, which increased the level of safety concern for all involved. A second site 

used a shroud-collar on their UAV to mask the GPS signal and flew in an open area. They also had 

radar surveillance as a second source of telemetry and (as the test specified) wanted to see how 

closely the radar signal aligned with their GPS telemetry with and without the shroud. The third site 

flew into and out of a building. 

Test sites reported very different experiences with GPS degradation. At the second test site, the team 

reported that it was difficult for them to degrade their GPS signal during the flight tests, and they 

saw no noticeable difference in telemetry between using a collar on their GPS and having no shroud. 

However, the third test site reported that they had trouble staying in their volume because they 

experienced a large GPS drift. This resulted in the UAS going both rogue and non-conforming 

during their CNS 2 tests. They observed high horizontal dilution of precision (of GPS signal) 

(HDOP) values and high horizontal and vertical position variance and one PIC reported he lost GPS 

lock multiple times during the tests. Survey questions asked about the reliability and quality of the 

Global Navigation Satellite System (GNSS) back up source and users’ situation awareness during 

these maneuvers. Debrief prompts focused on use of the USS client and navigation data availability 

(Table 22). 

77 


 

 

 

 

 

there was not a human factors component to these tests. The discussions reported below are from the 

debriefs that followed some days of CNS 3 testing. Debrief prompts focused on the quality of 

telemetry, what information a crew can glean from a degraded feed and crew situation awareness 

(Table 23). 

Table 23. CNS 3: Discussion Topics and their Notional Categorization 

into Information Themes 

Debrief Topic Category Site 2 Site 3 Site 4 Site 5 

Information property 3 3 4 

Information meets user 1 

Operator status 1 

Methods 4 

Automation 7 5 

Concept 1 1 

7.6.1 Technology: RF Interference 

Crews at different test sites reported different levels of RF interference in their testing. Those at Test 

Site 4 said they saw no interference, although they noted that they have experienced interference 

before when using the 2.4 GHz band. Test Site 4 also saw a drop in LTE coverage, but the USS Op 

confirmed that the drop was the LTE on the RF not related to position updates. Test Site 2 reported 

they did not see any unexpected RF frequency interference, as they were very familiar with the area 

and knew where they would see a decrease in signal on their flight corridor. 

In general discussion, crews reported having RF interference problems (in the 2.4 GHz band) when 

they have been flying next to cell repeater towers and have found magnetic interference around 

metal towers and power lines. They described magnetic interference, noting that the GPS fades first, 

then the compass fails, and at that point flying gets “sketchy.” Overload, crews described as cyclical, 

as cell phone usage is higher at some times of the day. They discussed whether the 2.4 gigahertz 

band may suffer from being drowned out by Wi-Fi, but crews thought that the Wi-Fi signals coming 

out of a building are too weak to cause disruption. Crews noted that they are more concerned about 

RF interference than they are about loss of GPS. If they lose GPS, then the UAV has RTB 

commands in its software, so there is not a problem with getting the UAV back to the GCS, but with 

RF interference, you are not sure which commands are getting to the UAV and which are not. 

One test site discussed a few possible sources for RF interference with the on-site NASA 

representative. In addition to external interference, ownship payload or electronics equipment could 

potentially also cause issues. Even during this TCL3 testing, the crews cited jammers that caused 

unexpected GNSS interference as a concern that could affect their navigation capabilities. 

7.6.2 Automation: Issues 

Crews reported issues they had with their USS client during their flight testing. One test site had an 

issue where their flight geography closed abruptly. Their complaint was that the interface did not 

80 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

give them any warning before the closure, nor any information that told them why. Their second 

concern was that their system stopped reporting telemetry when the volume closed, which meant 

their vehicle disappeared from UTM. For later tests, they ensured that their system kept recording if 

the volume closed, but it still did not live-transmit to NASA. 

Two test sites had issues with their volume submissions being rejected by the UTM system. One test 

site did not know at the time of the debrief whether it was an issue they had with not sending data in 

the correct format or time limit, or whether it was an issue with UTM. It caused them to spend some 

amount of time looking through their USS client logs to find “hiccups” and developing a troubleshooting plan. The second site was aware that the issue was with UTM, and it was because two sites 

were trying to use the NASA server at the same time and the USSs were both in the other’s LUN. 

In addition, crews reported some connectivity issues related to moving locations, because the port 

connections between the USS client and UTM were not always maintained as they moved, so they 

had to recycle the connections. They said that to all the other USSs, they just blipped off their 

screens. 

One participant argued that the time constraint to the flight geographies should be removed. Flights 

are delayed all the time, so the UTM system should just recognize when a UAV takes off, 

automatically activate the volume and start sending telemetry, and should automatically close when 

the UAV lands. 

7.7 CNS Tests Summary 

Test sites were interested in these C2 TCL3 tests, as all their crews had experienced telemetry faults 

and weaknesses during past missions. However, apart from known locations, the C2 link was 

reported to be stable throughout the TCL3 tests. Awareness of C2 status and data health was 

generally high, often with team members tasked to specifically monitor system health. 

• Safety: Feedback indicated that operators were generally happy with the safety of 

their operations during CNS testing. 

• Technology: Some crews noted a lag in their C2 data. Moving forward, crews made 

suggestions for more direct vehicle to USS connections in an effort to speed up data 

arriving to the user from the sensors. 

• SA: Discussions focused on what operators need to know about other flights. 

Moving forward, operators suggested continued progress to identify what they need 

to know about other operators in UTM and providing that information. 

Discussions focused on the human-technology interface, not only what tools and displays were 

available in their GCS, but also what features they would like their USS clients to have. As well as 

clear and timely alerting, crews wanted to be aware of their telemetry strength, but not be distracted 

by it. They were concerned about being flooded with alert messages, both from their own UAS and 

from others, if issues internal to one UAS were broadcast to all others in the area. They were also 

concerned about privacy if they had to report out all issues with their systems. 

• Usability: Crews emphasized that C2 link displays needed to be simple, and could 

be situated on the USS client in this case. Moving forward, indication of the 

strength of the C2 connection might be added to USS displays (simple gauge only). 

• Distractions: Users were concerned about being flooded with messages if every lost 

81 
 

 

 

 

 

 

 

 

 

 

 

link was to be broadcast to the community. This led to a wider discussion of how 

much is “too much” for all types of alerting. Moving forward, crews suggested that 

the community specify some lower limits on what needs to be broadcast, e.g., “lost 

link” of one or two seconds does not need to be broadcast. 

• Privacy: Users do not want to share their data or information, but agree that some 

level of sharing has to happen to make UTM useful. Implications are, as stated 

earlier, that the community should specify lower limits on what needs to be 

broadcast, outlining information that must be shared and when. 

• Procedures: Reports from crews indicated that they had no issues with their CNS 

procedures, possibly as these seem to be more clearly specified for CNS loss than 

other areas. 

• Design: UTM users were able to “step on” each other to reserve airspace, some did 

this accidentally, others gamed the system to gain an advantage. Moving forward, is 

the recommendation that these loopholes should be identified and solutions 

considered. 

8. TCL3 Conclusions and Recommendations 

Six test sites participated in the TCL3 testing from March through May of 2018, flying a subset of 

20 tests with one of four foci—CNS, DAT, CON or SAA. The mainly qualitative data discussed 

above was collected by on-site and remote NASA researchers. The data consisted of end-of-day 

debriefs, end-of-day surveys, observer notes, and flight test information, submitted as part of the data 

management plan. The TCL3 flight tests were successful, and more was learned about operators, the 

information they need to fly within UTM, and procedural requirements in a TCL3 environment. The 

findings presented above complement data previously collected for TCL2nc (Spring 2017) and 

TCL2 (Reno, 2016). Test-sites, again, markedly improved their tools and organization with respect 

to UTM, but also identified further areas that would benefit from more consideration. 

Taking general account of the 280 individual survey responses and 40 debriefs with crews at test 

sites that reviewed the 831 test flights made, the feedback from participants was positive in a number 

of ways. Data from CNS and DAT tests were very helpful, and the tests went as expected. The tools 

investigated with the SAA tests were well-received by operators and helped with their overall 

situation awareness. CON tests were interesting and varied and generated a good deal of discussion 

at both a detailed level of flying during different events and at a broad concept-level. 

In addition to items already addressed from previous flight tests, certain themes consistently surfaced 

in many discussions regardless of the particular test site or specific test being discussed. These 

themes relate directly to one of the six overarching categories that were pre-defined (Table 4) by the 

AOL and guided the data collection process to inform the question: “What information is required to 

successfully fly in a UTM environment?” All of these themes together help us understand how UTM 

was used, what went well and should persist, and how the UTM experience can be improved. 

1. Information Properties 

• Alerting Information. Crews discussed UTM-specific alerts at length, including rogue states. 

They discussed aspects of alerting triggers, distance from target, priority operations, balancing 

alerting frequency between too often and too seldom, what information should be broadcast, 

82 
 

 

 

 

 

 

 

 

and procedures for responding to alerts. Moving forward, alert characteristics, including 

information required, and expected proper responses need to be further defined. 

• Usability. Under-developed user interfaces contributed to users not accessing UTM 

information, and instead falling back on LOS techniques to ensure the safety and compliance 

of their vehicles. Examples of this included on-screen clutter caused by too many consecutive 

and repeated alert messages, and clutter from multiple (and sometimes too-sensitive) track 

sources being displayed simultaneously. Moving forward, standards could be developed for 

displaying relevant information to increase the usefulness of information offered while 

engaging with the UTM environment. 

2. Information Meets User 

• Trust and Reliance. False alarms and/or inconsistent responses from their system led users to 

rely on UTM information less because they could not trust that everything relevant was being 

detected and reported. Moving forward, the reliability of detecting and reporting information 

should be improved in order to increase trust in the systems, and therefore user reliance and 

buy-in to UTM. 

• Transparency of Tool. Interfaces showing UTM information did not always give enough 

information to the user about why automated actions took place. This led to longer times for 

the operators to diagnose issues and troubleshoot problems. Moving forward, tool transparency 

and information clarity need to be improved along with user training/experience on the system. 

3. Operator Status 

• Situation Awareness was influenced by the usability of the information presented by crews’ 

USS client, but also by how aware they were of what information was accessible. Operators 

who understood how UTM works, their part in the system, and how to access the information 

applicable to their tasks, had positive feedback about their UTM situation awareness. In 

addition to further GUI development, more training before the tests begin can help users 

understand how to utilize their tools and interpret the information meaningfully. 

• Workload reports were estimated at low or moderate, a very manageable level, even during 

flight tests that included off nominal events, suggesting that crews were able to manage the 

addition of the UTM role and the extra information load that brings within their teams. 

4. Methods 

• Procedures or “Rule Book.” Throughout the TCL3 testing, there was evidence that further 

development and research on procedures, standards, and/or recommendations are needed. For 

example, if two interacting vehicles were operating under different assumptions or guidelines, 

their responses to situations would not be predictable enough to support trust in the system. 

Defining proper responses for an array of theoretical situations, and then assessing them, can 

help users feel more comfortable with overall knowledge of their own and others’ operations. 

Example procedure/recommendation areas include: 

– Altitude reporting standardization (AGL/MSL) 

– Operational limits that define a rogue state and the reporting that goes along with 

that state change 

– Actions required in the event of a USS failure 

83 
 

 

 

 

 

 

 

 

– Actions required when interacting with a TFR 

– When and what information to show to other players 

– Guidelines for USS client displays 

– Cooperative planning between users of each USS to avoid “gaming” the system. 

5. Concept: Operator Buy-In 

• Training. As mentioned above, more training is advised to increase operators’ situation 

awareness, but training may also increase acceptance of the UTM concept and operators’ 

willingness to participate in this environment. Users with broad UTM knowledge were able to 

interact with the systems purposefully and provide a great deal of feedback regarding what 

they liked and disliked about both UTM concept and specific tools used. 

6. Automation 

• Level of Automation. There were many, varied, levels of automation available to the 

participants of TCL3 that affected their experience with, and opinions of, UTM. Some crews 

used tools that were nearly entirely automated, with very few tasks allocated to the human 

operator. Other crews had human operators manually flying their vehicles with little 

knowledge of the automation that was interacting with UTM on their behalf. Most crews used 

a system somewhere in between, making the TCL3 testing a good representation of what is 

anticipated in the future for automation integration. As would be expected, crews that relied on 

automation, and had everything go well, were more receptive to UTM, while crews that 

experienced issues or bugs, had a less positive response. Generally, operators liked the tools 

that were presented to them clearly and reliably. 

• Tools. Although tools, like those for the SAA tests, were very well-received, participants 

suggested additional features and functions, which, if not already supported with the current 

configuration, may be helpful additions in the future. 

Example features and functions suggested include: 

– Better feedback on USS clients 

– Ability to categorize a UREP 

– Arrival alerts for UREPs 

– Recoverability from a rogue state 

– Dynamic volumes 

– Two or more vehicles sharing launch locations 

– Ability to compare overlapping volumes and make suggestions 

– Ability to add a delay to an accepted volume 

– Integrating USS clients and auto-flight software 

– Severity component to alert messages 

– Always reporting key information with an incoming conflict 

– Audible alerts 

– C2 link-health status indicators 

– Auto-switch to secondary C2 link 

– Redundant sources for critical information in tools 

– Safe landing area identification 

– Automated de-confliction 

– Predicted vehicle trajectories with closure rates 

84 
9. References 

Federal Aviation Administration (2018). Unmanned Aircraft System (UAS) Traffic Management 

(UTM), Concept of Operations v1.0, Washington, DC, May 2018. 

Kopardekar, P., Rios, J., Prevot, T., Johnson, M., Jung, J., & Robinson, J. E. I. (2016). Unmanned 

Aircraft System Traffic Management (UTM) Concept of Operations, AIAA Aviation, 

Technology, Integration, and Operations Conference. 

Martin, L., Wolter, C., Gomez, A., & Mercer, J. (2018). TCL 2 National Campaign Human Factors 

Brief, NASA/TM-2018-219901, NASA Ames Research Center, Moffett Field, CA. 

Rios, J. (2017). UTM TCL3 Statement of Work, 27 July, 2017. 

Rios, J., Mercer, J. & Martin, L. (under revision, 2018). Use of UAS reports during TCL3 field 

testing, NASA/ TM, NASA Ames Research Center, Moffett Field, CA. 

85 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 1: Days of flying for TCL3 

Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 

5-Mar 

6-Mar 

7-Mar 

8-Mar 

9-Mar 

22-Mar 

27-Mar 

29-Mar 

3-Apr 

5-Apr 

6-Apr 

9-Apr 

10-Apr 

11-Apr 

12-Apr 

15-Apr 

16-Apr 

17-Apr 

18-Apr 

19-Apr 

23-Apr 

24-Apr 

25-Apr 

26-Apr 

27-Apr 

4-May 

7-May 

9-May 

10-May 

11-May 

15-May simulation only 

16-May 

17-May 

18-May 

23-May 

25-May 

30-May 

Shakedown days 1 1 4 0 3 1 

Test days 11 4 6 9 13 8 

Total days in 

field 

12 5 10 9 16 8 

Key: orange background = a shakedown day; green background = a data-collection day; blue 

background = a simulation only day 

NOTE: one site retested some of their data collection in the lab – hence there were 51 days of 

testing for data collection with 50 of these flying vehicles in the field and one simulating all flights 

in the lab. 

86 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 2: Crew Size and Composition 

GCS and flight crew information. 

Most 

common 

number of 

GCS 

Site 1 

Up to 5 

Site 2 

4 

Site 3 

2 (note TO site is 

different, there 

was 1 of these) 

Site 4 

2 

Site 5 

1 

Site 6 

5 

Number of 

crews flying 

Up to 5 3 real, 1 

simulated 

2 3 2 5 

Most 

common 

number of 

personnel in a 

flight crew 

3 

(not colocated) 

3-6 

(co-located) 

6 

(not co-located) 

2-3 

(colocated) 

4 

(not co-located) 

1 to 3 

(co-located for 

some crews) 

Common 

crew 

positions 

GCS op, 

USS client 

op, PIC, 

engineer, 

VO 

GCS op, 

USS client 

op, USS 

manager, 

PIC 

PIC & GCSO, 

VOs 

PIC, 

GCS/US 

Sop 

GCSO PIC, 

VOs = 

shared by 

both crews 

GCSO/PIC, 

USS Op, 

VO/ safety 

rep (other 

configuration 

s were used) 

Additional 

crew 

positions 

test 

manager 

Integration 

manager, 

engineer 

Additional 

collocated = 

UTM specialist, 

data engineer and 

at an alternate 

location: UAS ID 

field personnel = 

shared by both 

crews 

Additional = 

UTM 

operator 

flight 

manager, 

test 

coordinator, 

package 

loader 

87 
 

 

 

 

 

 

 

Appendix 3: Vehicles Flown in Flight Test 

Number of 

UAS 

models 

Site 1 

6 

Site 2 

3 

Site 3 

2 

Site 4 

4 

Site 5 

5 

Site 6 

8 

Type of 6 multi- 1 multi- 2 multi- 1 multi-rotor 3 multi-rotor 1 hybrid 

UAS rotor rotor rotor 2 fixed-wing 2 fixed-wing 5 multi-rotor 

flown 1 fixedwing, 1 

hybrid 

1 helicopter 2 fixed-wing 

Model of S1000, VTOL S1000, F7200, UASUSA eBee, Aero, 

UAS S900, Savant, Matrice100 F6500, Tempest, Hummingbird, 

F450, NAV-X, T- vapor, F7200, Solo, Intel Aero, 

Matrice master Sharper A6, Matrice 600, Inspire, 

100, AR-180 S1000, Eagle 

inspire, XF, Phantom 

ptarmigan 3 

88 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 4: List of Flight Tests 

Test 

identifier Title 

CNS Communication, Navigation, and Surveillance 

CNS1 Maintaining control of UA with Redundant C2 

CNS2 Remaining within Flight Geography using GNSS Navigation 

CNS3 RF Interference Baseline Monitoring 

SAA Sense and Avoid 

SAA1 Air to Air Conflict Mitigation Cooperative Technology for UAS-UAS

Interaction 

SAA2 Air to Air Conflict Mitigation Cooperative Technology for UAS-Manned

Interaction 

SAA3 Air to Air Conflict Mitigation Non-Cooperative Technology for UASManned Interaction 

SAA4 Air to Ground Conflict Mitigation Non-Cooperative Technology for UASManned Interaction 

SAA5 System Level Assessment and Off nominal conditions 

SAA6 Air to Ground Remote Identification and interoperability with automobiles

using Cooperative Technology 

DAT Data and Information Exchange 

DAT1 End-to-end UREP 

DAT2 FIMS failover 

DAT3 USS failover 

DAT4 UAS ID 

DAT5 USS-USS Negotiation 

DAT6 Wx Service 

CON Concepts 

CON1 BVLOS Landing 

CON2 Contingency Initiation 

CON4 Multiple TCL-2/3 operations for a sustained period 

CON5 FIMS/USS interaction when vehicle heads towards Controlled or

unauthorized airspace 

89 
 

 

 

 

 

 

 

Appendix 5: Debriefs Conducted 

Debrief Interviews: Location and test discussed. 

Test Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 ALL 

DAT1 1 • • • • 1 

DAT2 •

DAT3 • • •

DAT4 2 1 3 

DAT5 1 • 1 • • 2 

DAT6 •

DAT multi 

test 

1 1 1 2 5 

CON 1 1 1 

CON 2 • 1 • 1 

CON 4 1 • 1 1 1 • 4 

CON 5 2 1 3 

CON multi 

test 1 2 3 

SAA 1 •

SAA 2 1 1 

SAA 3 1 1 

SAA 4 1 1 

SAA 5 

SAA 6 •

SAA multi 

test 1 1 

CNS 1 • 1 1 2 

CNS 2 1 • 1 2 

CNS 3 • 2 2 4 

CNS multi 

test 1 1 

•: 

Check marks denote that this test was covered in a debrief that discussed multiple tests. The number 

of multi-test/ multi-topic debriefs is listed in the “multi-test” rows. 

90 
 

 

Appendix 6: Flights Made by Test Sites to Fulfill each Test 

Flight activity by Test (includes GUFIs with or without valid positions). 

NASA Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 ALL 

CNS1 21 8 19 15 1 64 

CNS2 6 4 4 14 

CNS3 3 21 25 6 55 

CNS 27 15 40 40 11 133 

SAA1 8 8 

SAA2 8 4 12 

SAA3 2 5 7 

SAA4 3 6 9 

SAA5 2 2 4 

SAA6 1 1 

SAA 11 14 16 41 

DAT1 12 8 6 26 

DAT2 9 4 1 14 

DAT3 3 21 6 5 3 38 

DAT4 14 5 19 

DAT5 13 38 17 7 53 128 

DAT6 10 10 

DAT 16 90 14 40 18 57 235 

CON1 4 4 5 13 

CON2 10 8 14 32 

CON4 5 34 25 2 250 316 

CON5 12 18 1 6 3 40 

CON 27 64 1 4 31 5 269 401 

ALL 43 102 106 58 111 48 342 810 

91 
 

 

 

 

 

 

 

 

Appendix 7: Examples of Topics Discussed in DAT Interviews with

Flight Crews 

A selection of excerpts from field debriefs with test site crews for DAT tests and the corresponding 

topic category. 

DAT 1 

Information: 

Usability of 

information 

7 We need to see the heading, speed and altitude information in 

an easier to read format—during our time in the air we did not

see any of the other aircraft, real or simulated on the screen. If 

the system is going to deconflict two aircraft it needs to be

very clear as to what we need to do and what the other aircraft

has been told to do. This could be shown as a warning box and 

a suggested path/heading with countdown timer. 

Information: 

Accuracy 

9 The distractions and inaccuracies of the current USS cause 

unneeded confusion in the GCS 

Methods: 

Procedures 

4 Our flight operations with our GCS worked according to plan. 

No surprises—the addition of the USS was distracting, our 

USS manager worked to insulate us (the flight team) from

having to deal with its inconsistencies. We attempted to use

the USS in conjunction with our GCS but it became a

workload safety issue so instead we received real time reports

from our USS Manager. 

Automation: 

Functionality 

4 So the system was doing automatic UREPs for the weather, it

was doing automatic UREPs for traffic that was ID’d from

different sensors, and then the manual UREPs were submitted 

by basic crew. 

DAT 3 

Information: 

Usability of 

information 

4 The only way in our system you wouldn’t know it’s not fully 

functioning is that on our big map display your own stuff 

doesn’t get updated anymore or you don’t get any updates. 

Just a static screen. I think that’s a function of the map. 

Information meets 

UTM: Decision 

making 

3 If you had just the UTM part of it go out, you might still have

access to things like the TFMS feed so you’d know where

other manned traffic is. You’d still have your links into, you 

know, like supplementary ground radars, ADSB receivers, 

anything else. The person who’s in command of the mission 

would have to make decisions about “what do I know” and 

“what does that mean for me right now?” 

Operator

experiences:

Situation awareness 

7 When you come back up, it’s not like instantly you see it, 

right? You need to wait for a little bit for things to catch up, 

‘cause there’s nothing that, um, necessarily tells you right

away what messages you’ve missed. 

Methods: Rules 10 Not if already in the air. If in planning it would let you submit

the plan. If flying under normal 107, no need for UTM so can 

continue without UTM as a normal 107 operation. If flying 

92 
 

 

 

 

 

 

 

 

 

with UTM is a requirement for your operation (say a

requirement of your 107X waiver) then you would be unable

to fly. 

Automation: Design

of system 

Concept: 

Requirements 

5 

4 

Because when the network link would come back up, our USS

because it didn’t know it failed didn’t know it missed 

anything. 

Some of the scalability issues with positions reports for USSs

that are unresponsive. So, if you have to reach out to every 

other USS who’s subscribed in your positions, once every 

second, for every operation you have, that’s already a decent

amount of independent HGTD (?) calls to be making. If one of 

the USSs is out, every single one of those to that USS for 

every single operation you’re sending every second is gonna

be timing out. So, you add a lot more strain on your system

waiting for the time-outs for every single position report for 

the operation. 

I didn’t really have good visibility into what the ping station, 

which was mounted on top, searches as far as the receiver. I 

DAT 4 

Operator

experiences:

Situation 

Awareness 

Methods: Roles 

15 

8 

didn’t have good visibility to what it was doing, um, I was

only looking at the hand-held receiver piece of it. Um, and the, 

yeah – like I said, I could – it was detected long before I could 

see or hear it. But again, we were behind trees and a water 

tower and all that other stuff, but, it was kind of good that I 

could see it quite a while. 

You can have a team of operators and a, because for example, 

they found that the workload was too high for the command 

and control operator. They separate roles but then shared a

similar space so they can communicate better. 

Automation: Design re very much focused on automation being the answer to 

of system 

Concept: Privacy 

DAT 5 

12 We'

safety rather than additional knowledge that actually told us to 

take data away because they don't need it because it doesn't

make a decision. 

11 And for us to have balance between transparency and privacy 

is the distance to be able to handle it on demand, not store

data, feed it in a particular way that's needed for that. 

From what I observed I think a key piece of information is

some context or information not really the intent that can be

embedded in the operation or put somewhere that can be

called for a negotiation so that can inform the mission. 

There is a lot of trust in the system, there’s trust that when 

you’re asking for a negotiation that you actually need it, and 

there’s trust from the other side that when I ask you for a

negotiation, that we’re trusting that you’ll like actually try, and 

we’re trusting that what you get back is actually valid and deInformation: 

Usefulness of 

information 

Information: Trust 

8 

10 

conflicted. 

93 
 

 

Operator

experiences:

Situation 

Awareness 

20 Everyone likes having the display with ADS-B because it’s

just another level of situational awareness. If this is the display 

they’re using for situational awareness. 

Methods: 

Procedures 

3 And so, you know, like “I got battery, and I got everybody 

sitting here,” so there does need to be some rules of the road, I 

believe, so that if it’s X-amount of time before the window

then you’re locked out, or you know, there does need to be

consideration that there are certain times where you don’t want

your plan pulled out from under you. 

Automation: Design

of system 

21 I think it's important to have APIs that scale across multiple

interfaces. 

Concept 

DAT 6 

3 We think it's important to have small operators be able to 

participate in a USS in an easy way because that's the way we 

keep the whole of the system safe. 

Yeah, you actually had to click each one and then see where it

came from or what type it was, but it was just essentially a list

Information: 

Usability/Usefulness 

3 

of time stamps. 

Information meets 

UTM: Input 

demands 

3 We’re trying to make it so you had to fill out the least amount

of information when you had it...Which is good because then 

it also can be more accurate, right, because you already – well, 

if you did like a weather report in the air, but like if we did the

ground one, the aircraft might be over there, you know . . . 

Methods: 

Procedures 

4 We have a weather overlay of ER and apps on our screen but

our pilots also have their own checklist and I think that once

they are done getting their route, they still go through besides

the USS screen. They have their own checklist and they would 

also get it on the plane. Commercial airplane pilots already 

have their own source of weather data that they are used to 

getting it from and at the moment they trust. 

Automation: Data 

rates 

2 We’re very much focused on automation being the answer to 

safety rather than additional knowledge that actually told us to 

take data away because they don't need it because it doesn't

make a decision. 

94 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 8: Examples of Notes from DAT Observations 

A selection of excerpts from debriefings and NASA representatives’ field observation notes and 

their corresponding topics. 

DAT 1 

Information: 

Transparency of 

tool 

It was easy to send UREPs but the search for new UREPs was not intuitive. 

Information 

meets UTM: 

Quality of alerts 

UREP received, but only see Air temp, wind speed. No aircraft sighting info. 

Automation: 

Design of system 

UREPS are .JSON files that come through. Can display graphical information as well as 

text. Issues with the [USS] “listening” for the message every # of ms, and then ignoring 

messages in the past. And if [USS] sends the message in between the “listening” time then 

they don’t get it. 

DAT 3 

Methods: 

Procedures 

Q: How did it “fail” and how did each crewmember react? 

A: We simulated some USS failure by resetting [USS1] while in-flight. In parallel, [another 

crew] was flying some mission. Crewmembers reacted by checking the expected “503 

service unavailable” status codes returned by [USS1] during the USS failover. 

Information 

Required/Desired 

Q: What information do you need when your USS fails?

A: We just need our USS to restart and to catch up with what happened in the UTM while it

was failing. 

Automation: 

Design of system 

Ultimately, USS should be fully automated. We should however persist USS failure events 

so that we can inspect what happened. 

Information: 

Required/Desired 

Additionally, there currently was no alert in the GCS for the pilot to know that the GCS is 

down. 

Information: 

Required/Desired 

Q: What information do you need when your USS fails?

A: The pilots need to know what that failure means for them. Should they land or should

they keep flying on their flight plan? Should they activate a contingency? The GCS operator

needs to know the failure (on his end, on the USS end?) and if there are any debugging steps

s/he should take. Also what s/he should communicate to the pilot (if anything). 

Methods: 

Description of 

operator roles 

Q: What were the individual responsibilities of each person? 

• USS client operator: Monitor the airspace in real time and communicate any USS

alerts to the GCS operator. 

• USS manager: Coordinate testing with other USSs so that we can execute the tests. 

• GCS operator: Communicate with the pilot about flight plan approval and

connectivity and telemetry. Monitor conformance. Communicate with USS

operator if needed. 

Operator status:

Workload 

USS op was also in charge of slack, has to hand off task to mission command because of

workload. 

DAT 4 

Operator status:

Workload 

Both crews are hurrying. 

Other [IR beacon detection] device only detects with an active flight plan. 

Information: 

Usefulness 

Flight crew thinks remote ID is useful in real time to help a flight crew have better

awareness of the UFO (i.e. looking up the vehicle type is useful, looking up the created 

information the registered owner is not). 

95 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

DAT 5 

Information: 

Timeliness 

Q: Did you receive the negotiation information with enough time and enough information to 

safely mitigate the conflict?

A: Yes. The USS and the GCS interfaces notified users immediately if their proposed route

was in conflict. If incoming route from another USS were in conflict, the USS screens

would notify the user. 

Automation: 

Design of system 

Q: Do you think the USS-USS negotiation strategy was effective?

A: Somewhat. Because some USSs took up the entire airport instead of trajectory-based 

reservation, it was hard to find alternatives that weren’t in conflict. Also, the manual nature 

of negotiation made it cumbersome to figure out alternatives. It was much easier to just call 

and talk. 

Information: 

Usability 

Q: If you were instructed to make changes to your plan because of a higher priority mission 

coming into your airspace, did you have enough awareness or were you provided with 

enough information of the surrounding operations to do this safely?

A: Yes. There was a map with pictures showing where the conflicts existed and where other

operations were and it allowed us to visually figure out how to re-plan the flights. 

Automation: 

Design of system 

A NASA representative noted that the highly autonomous nature of the operations made the

crew irrelevant, or not existent. 

Methods: 

Procedures 

One site flew three versions of this test. To begin, the second operation’s USS initiated

negotiations with the other USS and their request to replan was granted. In another scenario,

they requested a manual clearance, which was denied, and then a third scenario, where they 

requested a manual clearance, it was granted, and the operators continued with

manual/visual deconfliction. Our NASA representative asked, “What if pilots or GCSs 

weren’t close enough to each other to verbally chat. Would that change a response to a 

request for clearance to fly in the same airspace?” 

96 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 9: Concerning Nearby Operations 

Nearby operations that were the most concerning to the survey respondent, n = 18. 

Test 

Site Theme Comment about Most Concerning Operations 

1 BVLOS BVLOS landing flight from the S1000 

6 BVLOS Simulated BVLOS 

6 BVLOS BVLOS operations due to the large area and no single observer. 

3 No 

UTM 

The last flight of the day was an intruder flight, and not within the UTM system. We could not 

actively determine the bounds of the flight, alt, heading, etc. This increases awareness of our

operations because the aircraft is less predictable without the assistance of UTM. 

4 Altitude The other two operators were at altitudes of 100 ft above and below our location. The altitudes

planned by each operator were done by AGL. This ended up causing some concern from a fixed 

wing hovering close you our take off location. Since it had launch at the lake bed which is 

approximately 50 ft lower than our location, it was only about 50 to 60 ft above where we were 

going to climb to. The other operators appeared lower than us since they were launching from

another ground location. If all operators were required to figure out their MSL location, then the

altitude separation and future segmented operations would be much more seamless to plan and 

accomplish. 

6 Altitude The nearby operation most concerning were XX because their flight path was right above our 

launch location. 

4 Overreserve 

The difficulty was the operators were reserving large airspace and it was difficult to fly multiple 

aircraft. 2 of 3 clients could not handle multi-segment ops. That would have reduced the conflict 

although not eliminated it. 

4 Over 

reserve 

Those who just grab the entire base for the whole day as their op when in fact they just need a

small part for a short time. 

4 No 

reports 

Last with an aircraft not reporting. 

4 No 

report 

Last for flight watching for the non-ID aircraft. 

4 No 

report 

During the last flight another aircraft was orbiting nearby and was not broadcasting its position in

the UTM map. 

4 Rogue

Close by 

Rogue ops and ones in close proximity are the most concerning as they are most likely to affect me. 

5 Close by Ops in my Landing zone 

6 Close by Other XX operators, because we launch very close to each other so need to pay attention during 

launch and land. 

6 Close by Most concerning were operations adjacent to my launch/landing zone.

Reason being due to ascending and descending through their flight level after launch and before 

landing.

There was good separation and no need for concern, but in my opinion required heightened 

situational awareness despite USS automated deconfliction. 

6 Public Non-participants in area. 

97 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 10: Examples of Topics Discussed in CON Interviews with 

Flight Crews 

A selection of excerpts from field debriefs with test site crews for CON tests and the corresponding 

topic category. 

CON 1 

Information: 

Quality of alerts 3 

There are like 2 uses for those sensors. One is so that the aircraft does not 

run into something in the building and the second is also situational 

awareness. 

Information meets 

user: Speed of data 

transfer 

6 I think that is really critical as fast as you can tell us where is and how fast

and what it is doing so we can take some manual action to avoid it. 

Operator

experience:

Situation Awareness 

6 

We have live tracking we know where it is and we are confident we know

where it is as long as your get positive affirmation that there is a person

somehow responsive and I think that is the whole idea of the UTM client

make sure still alive and functional. 

Method 1 

Automation: Status 

7 

Like last year when we did dynamic replan we trying to do specialized

cases the same points potentially we didn’t run across but the fact that if

you go rogue you can never go back and it’s nice to have alerts but we also 

can go back. 

Concept: Design of 

concept 4 

So, you can have another aircraft following a little ways behind you, your 

track is closing behind you so he is good to fly behind you but your next 

track would involve going backward so you would not get clearance to [go] 

there. 

CON 2 

Information: 

Sharing information 8 

The main information for us in terms of sharing information amongst USS

is that if an operation has gone rogue to be alerted to that, that, or be alerted

to that event so that we can you know, inform or, or educate any of the, 

flights that are handled under, under our USS. 

Information meets 

user: Confidence in 

information 

14 

If I, in the event of a non-planned emergency where I need to, I know the 

aircraft is returning and I'm concerned I'm a potentially in the way, so I

need to have a way that I can look at it at glance and see that, uh, that I

need to clear or that I need to change what, what I'm doing. 

Operator

experience:

Situation Awareness 

4 

I think that positions are very important. That tells you exactly where on

the aircraft is that and allows you to make sure that you maintain your

separation from each other. 

Method: Procedures 

for operations 5 

We have to generate airspace for it, which we do. That airspace is now at

an increased priority. So it always made any other vehicle that was maybe 

nearby, would make it rogue because you were conflicting with a priority 

airspace. So just because I'm a guy who's misbehaving...is kicking 

everybody else and making them go rogue, is that really a fair thing to do? 

CON 4 

Information: 

Usefulness 

25 My map is like super open and I know it’s not cluttered, so planning a map

for emissions for me was super easy until the negotiations started and I

couldn’t see my own plan. ‘Cause I couldn’t click on my own plan. 

Information meets 

user: Desired 

information 

14 I would like back from the USS about how close to compliance I am. So 

that if I'm slow I could speed the craft up and get back in compliance with 

my plan or if I’m going to fast I could slow down. 

98 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Operator

Experience: SA 

(lack of) 

11 We couldn’t communicate with anybody, we had no contact information.

So, basically, we would put a plan it, we’d get rejected, we’d change it, and 

we could see what others had planned or tried to plan, we couldn’t tell

what, so then we’d just make a boundary that would intersect with the 

fewest lines ... We could see other operations, but like we were in very

small areas, so we only grabbed just small chunks of pie, and that seemed

to work better. 

Method: Gaming 

the system 

23 The other thing is, cost ... So, if we want to go and inspect the water tower 

and we’re making money here, we have to do this, and there’s somebody 

that just has a huge area ... in real life, we would have just flown that 107

and we would have just avoided it [the other vehicle]. 

Automation: Display 

conventions 

25 All users should kind of appear the same to the UTM. The UTM shouldn't 

necessarily know if it's a human operating or an autonomous system right,

it’s getting the same information. And then on the other side it should be

given the same information out, and then however the user as sophisticated 

as they want to be with it, they can do. 

Concept: Usage

rules 

CON 5 

15 The system is really vulnerable to somebody just being like “I want this 

whole county for all day.” Like you could easily have done that. Be like

“Oh, I guess that’s what we do!” And so, some kind of incentivization to be

as efficient with your flight plan as possible is good. 

I was pretty impressed by the notifications on just our restricted airspace.

Um, just with the proximity warning and then actually crossing it was dead

on and seemed to work pretty great. 

Information: 

Usefulness 

29 

Information meets 

user: Alerting 

10 Yeah, a reminder while you are submitting the flight plan would maybe be

nice, but I don’t need the 30-second reminder the whole time because I 

already knew it. 

Operator

experience:

Situation Awareness 

18 Like you said, you announce that there’s a rogue aircraft, but you won’t tell 

people where. I guess if it was me operating, if you announced that there’s

a rogue aircraft, I want to know where it’s at. If it’s not going to be a factor 

for me, then I don’t even want to know it’s rogue, because you hear that 

and you’re constantly.........just announcing rogue isn’t necessarily 

beneficial. 

Method: Procedures 

for event 

5 We have a two-minute, an arbitrary but a two minute period for you to 

vacate the airspace. So, um, so there's actually kind of hard to tell, but

there's actually two volumes. Is the larger volume, the original one still 

stays in place and, but it's end time gets shortened, but it's still two, you still 

have two minutes of time in there. 

Automation: 

Functionality 

13 The goal, you know, a lot of folks want full automation, so if you do more

of that, we've got to add more smarts into the system to actually be able to 

make those, I'd say pilot kind of choices. 

Concept – Airspace 

rules 

17 You can’t cross out – so, that goes back to how big of a performance

geography do you need, because you can’t – even if the pilot crosses out of

their protective geography, what’s kind of reserved for them, that impacts

others. So, just saying “Hey, the pilot understands and is going to try to 

rectify,” that could impact somebody else so, that part, you know .... That’s 

just gonna lead to people requesting a larger amount of airspace for a

longer amount of time. 

99 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 11: Examples of Notes from CON Observations 

A selection of excerpts from NASA representatives’ field observation notes and the corresponding 

topic category. 

CON 1 

Methods: 

Description of 

operator roles 

The PIC, the GCSO, the UTM Operator, and the Mission Manager were all located [inside].

Outside, there were always VOs, and for CON1 flights, a RC safety pilot was also stationed

with them. 

Methods: 

Procedures for 

flight/event 

CON1 for [the test site] includes taking off from within a hangar using a laser rangefinder to

navigate through guided waypoints, exiting the hangar, loitering and switching to Auto mode,

flying to a further BVLOS location (behind trees in the triangle at the compass rose), landing

in auto mode, then taking off again automatically, traversing back to the hangar, and landing 

inside in that same automatic-type mode. PIC, GCSO, and UTM Op are all co-located, 

BVLOS of the operation. For CON1 only, there is an RC safety pilot at the original takeoff 

location/2nd landing location to take control if necessary. 

Information: 

Saliency,

Other: 

Operator buyin 

UTM Operator is the only/first person to recognize that during a power cycle, the position

reporting would stop and they’d go rogue. The sole concern of the PIC GCSO is the safety of 

the vehicle. 

Information: 

Transparency 

of fool 

Terrain data is still missing. Crew can’t yet figure out the problem. 

Information: 

Reliance,

Trust,

Confidence in 

maneuver 

GCSO begins glancing at video on the return @ waypoint 4, PIC starts @ waypoint 5. GCSO 

starts tracking the video (80% of his focus) at 70 feet from the hangar doors, the PIC starts 

tracking the video (with 100% focus) at 80 feet from the hangar doors, and also with his mouse

hovering over “LAND.” 

Information: 

Saliency, 

Reliance 

No one is worrying about [the USS client}. They’ve never had a state problem that I’ve seen so 

far. 

CON 2 

Information: 

Usefulness, 

Interpretability 

Not only are there visual tracks, but they get text notifications that persist on the screen for a 

period of time. Because of number of false notifications, the text was covering the majority of

the display. Rendering it useless. 

Other: 

Confounds 

The secondary ground radar was having trouble with the birds in the area. They system was 

smart enough to exclude individual birds, but flocks of birds were large enough to be

considered a target. This made it difficult to calibrate the radar. Mission manager/UTM op 

decided to calibrate the radar using [a] vehicle. They had the pilot fly various encounters to

calibrate. They noticed that the [vehicle] would show up accurately but then unexpectedly 

show a negative value for its altitude. Crew continued after a little bit of calibration/code work 

Automation: 

Design of 

system or

hardware 

XXX client isn’t processing the data very well (fusing the tracks). Tempest crew can’t use tool 

successfully. 

Information – 

Transparency 

of fool 

USS to USS notifications for contingency don’t include a code for “altitude contingency,” so 

they used the loiter identifier. Execute same procedures as above, but change altitude by 100ft 

upon intruder notification instead of loiter. Since, vehicles were already altitude separated, 

100ft change didn’t seem to affect much in the DAA software. 

100 
 

 

 

 

 

 

 

Operator

status: 

Situation 

Awareness 

This led to discussion on how the DAA software works. There is a “preventative DAA” and 

not and the crews admit they don’t fully understand the state changes for the DAA. They know 

it has to do with range proximity, but the state changes are so quick and erratic that it’s hard to

nail down cause and effect. 

Automation: 

Functions 

XXX client display has 4 levels of “vehicle interaction”. 

o Clear of conflict 

o Preventative 

o Corrective 

o Warning 

CON 4 

Information: 

Timeliness 

[Vehicle] went [non-conforming] because it switched radio links. 

101 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 12: Examples of Topics Discussed in SAA Interviews with 

Flight Crews 

Examples of quotes from debriefs with test site crews after a day of flying and the corresponding 

topic category. 

SAA 1 SAA 6 

Information: 

Quality of alerts 

9 There is, we didn't get to hear it yet though because the... at XXX they,

they enabled it to before we had a chance to test it in that flight. Didn't get

to hear it on the ground wherever we were seeing the alerts on the ground. 

Information meets 

use: Transparency 

of tool 

17 Can we design UTM to be almost completely transparent to the user such

as just during the course of them flight planning, plotting their waypoints? 

Operator

experience:

Situation Awareness 

11 It's when you have it out there by itself and no one's got eyes on it and then

you've got another object coming out at high speed somewhere near its path 

that you really need all that extra situational awareness. 

Method: Procedures 

for flight/event 

3 That kind of USB permission gain kind of jars that workflow and provides

a disincentive to use this third party app when it's a lot of overhead just to 

get at the basic level of functionality working. 

Automation: 

Functionality 

17 If you can leverage that existing infrastructure. I'm just looking at the

interstate highway system for one and if we have these existing nodes and

the ability of VDOT to use the aircraft that can then connect, communicate 

with their DSRC network in a mobile manner I think opens up a lot of

opportunities there. 

Concept: Equipage

requirements 

21 I don't think we're there yet with the regulatory space. I don't know that 

we're going there at all. I think we are really what we're looking at is

equipage requirements depending on the level of access and sophistication 

and risks that your missions require, so for BVLOS operations and opposite

for people and things, maybe some of this would be, but we're definitely

not headed down the path of imposing these equipage requirements on rc or

just general user. 

SAA 2 

Information: 

Reliability/ accuracy 

of information 

17 We were seeing ADSB aircraft from XXX which is seems to be local and

some of that is, seems to be there's a discrepancy and some of the ADSB

timestamps. So like when I view a track list from the Ping station, I'll see

like 15 aircraft and I'll look at them and they have like all these timestamps

within like maybe a few hours. It was like four, maybe four hours or so, 

which doesn't really make sense. They should be like, if I log into that and

they should all be like, I just received that timestamp. The seems like some 

of these ADSB out units from different aircraft are not programmed 

correctly on their timestamp. 

Information meets 

user: 

Appropriateness of 

maneuver 

10 He came straight at me so I couldn't do the RTL because it will cross its 

path....I saw the aircraft continue to move.... like that's unacceptable. So

then the next movement is down and it so much easier for me to go down

than it is for uh, for him to go down. 

Operator

experience:

Situation Awareness 

43 But this is also the rabbit hole we went down when we were talking by the

tables that aren't for your, uh, the delay in this, this particular system. The

delay to getting the information to pilot was such that you couldn't react on 

it as a sense and avoid use. 

Method: Procedures 

for flight/event 

10 What we're talking about is the response. Oh yeah. But it would be a TCAS 

like response of just the UAS. The reason the FAA doesn't want us to put 

TCAS on board UAS is because that would require manned aviation to 

102 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

avoid us. If there was a conflict, there would be both. Both aircraft are 

required to avoid. 

Automation: Design

of system 

14 It's still that hierarchy of mobility, I guess where we have our mobile

devices that we get out of the way. 

SAA 3 

Information: 

Quality of alerting 

21 ... that's one thing that was missing on our display ... I think we needed 

more audible alerting ... I don't recall hearing any audibles. I mean, you 

know, we were... It's not a noisy environment but not a quiet environment 

either. 

Information meets 

user: Confidence in 

information 

15 So it's something that could add like just visual colors or visual indicators

of, of the likes of the track either with trails or sizes of the, of the warning 

of some way, right? But visualizing the confidence of which we report

actually in the radar packet. The confidence we have in the track that could

be visually kind of identified on the screen in some way that helps the pilot

kind of know that. 

Operator

experience:

Workload 

4 That's, that's really the big, um, I want to say not real world piece of this 

whole concept right now that the pilots are dealing with, is that they are

dealing with two pieces of software to manage UTM functionality and then 

to manage is piloting and the work load involved there is really at 

unacceptable levels. 

Method: Procedures 

for flight/event 

3 Another thing that we had to kind of work around for this test with the

current, uh, USS configuration was, um, that we only had the ability to 

define one radius and we went ahead and assume that that was our, um,

well, I guess in different tests we actually made different assumptions, but,

you know, ... there's several layers of alerting, right? 

Automation: Design

of system 

15 The other weakness we saw quite a bit today with the dependence on that 

server and its livelihood, and as you saw, we had a number of server fault

then, you know, maybe that's the problem that can be solved, but the

question remains is it a barrier that we want to tolerate? Um, I think there's

a lot of functionality that can be gained with a cloud based systems ... 

SAA 4 

Information: 

Quality of alerts

saliency of 

information 

15 ... a green, amber, red, a display. It was the icon of the aircraft. It has like a 

range warning system, but it doesn't throw up any popups. It doesn't waive 

any flags or anything. You just see the aircraft icons go from green to 

amber to red. 

Information meets 

user: Information 

desired 

6 If I was the cooperative bird, I would want to know basically what, uh,

what any radar operator would want to know. I want to where the aircraft,

or where the target is. Uh, is it approaching me? How fast is it going? Is it 

at the same altitude? 

Operator

experience:

Situation Awareness 

5 ... it's the pilot's duty to give way. So I would start if I was just hand flying

this thing and I was not prepared for this. Then the instant that I saw that 

track was going to crossover into my amber area. Then I would start, I 

would attempt to maneuver and get them to keep them outside the bubble. 

Method: Procedures 

for flight/event 

9 The way that I've written this to, uh, for the manual side in order to

compensate for the lack of the timing that a computer would give us is a,

the wording is that when an aircraft or target is inside the warning area, it's 

the pilot's duty to give way. 

Automation: 

Functionality 

13 Because you have to have that capability. When the aircraft loses link, you

don't have the capability of the aircraft avoiding on its own. When you lose 

link, you have a rogue aircraft in the air, in the airspace, if it's a busy 

airspace and that's where we want to get to eventually that's a bad thing. So 

the aircraft has to be able to avoid the other aircraft. 

Concept – Equipage

requirements 

1 ... sense and avoid at all times, otherwise you don't get to play. 

103 
Appendix 13: Examples of Notes from SAA Observations 

A selection of excerpts from NASA representatives’ field observation notes and the corresponding 

topic category. 

SAA 1 

Information: 

Reliability, accuracy 

Can’t always see the second vehicle on the screen – some flights you can see both

vehicles in others you can’t. On a couple of occasions NASA reports that they can’t

see [vehicle 1] but we wonder whether it’s actually [vehicle 2] they can’t see. 

Information: 

Quality of alerts, 

Transparency of 

tool 

The ability to submit multiple operations under one volume. [Site 6] are not trying to 

do this but after a flight where there were lots of odd rogue messages but no alerting 

when the vehicles intruded on each other, ANRA said that [Site 6] had submitted 

both vehicles to the same operation that caused those effects because the vehicles 

were “cooperating”. XX couldn’t see why you would ever want to have the alerting 

completely off. 

Other: Operator

buy-in 

Not enough information on the USS for pilots to be able to fly by it, so although

they are looking at it during this week of testing they say they would not use it in the 

field. 

Method: Procedures 

for flight/event 

Crew was unclear what constituted a resolution maneuver in the SAA flight tests

and what defined that they had successfully completed the test. 

SAA 2 

Information: 

Quality of alerts, 

usefulness 

Auditory alerts were turned off 

• Brought up two excellent discussions: 1. Reason why they muted alerts

(would continually warn about its own ADSB tracks as an intruder) and the

concept of selectively muting alerts. 2. Refresh rate of displays vs actual

computational time of data needed to DAA. 

Information: 

Accuracy 

The UAV popped up immediately for the manned AC and he was able to locate and 

fly in the direction of the UAV. 

SAA 4 

Method: Procedures 

for flight/event,

automation 

functions 

Somewhere along their path, they would come within 50m of each other and trigger

an intruder alert. In an automated DAA, this intruder alert would trigger an 

automated response from the vehicle. Today however, it was a manual response to 

the intruder alert. 

Information: 

Usability/usefulness, 

intuitiveness 

The radar can see both aircraft and sends that data to mission planner, but mission

planner cannot discriminate between radar tracks belonging to the intruder from

those belonging to ownship, putting the pilot (when BVLOS) or automation in a

situation where they might try to “avoid themselves.” 

Information: 

Usefulness 

Successfully got radar displayed on GCS. This involved getting radar tracks,

converting them into "ADS-B like" tracks to get them into mission planner (GCS). 

SAA 6 

Information: 

Quality of alerts 

First pass at 20mph, we did get an alert on the [USS] display but no audio alert only 

the red volume a message appears. 

Information: 

Quality of alerts,

reliability 

Car makes a couple more passes, we can see alerting. Some of it is spurious as the 

car is nowhere near the UAV. 

General SAA 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

104 
 

 

 

 

 

 

Information: 

Usability, quality of 

alerts 

Think an audio component would be useful – having to look at the screen to see

alerts makes it not useful, for 2 reasons: 1) you won’t know when alerts will arrive 

when you’re flying real world missions, so you won’t be looking at the screen

waiting for them; 2) having to look at the alerts means you take your eye off your

vehicle. 

Information: 

Quality of alerts 

For one test with the [Vehicle 1] intruding on the [Vehicle 2] we did not get any 

alerting. 

105 
 

 

Appendix 14: CNS Capability on UAV 

5 

8 

3 

9 

7 

3 

Site 1: auto switch Site 2: auto switch Site 3: auto switch 

Site 4: auto switch Site 5: auto switch No, not automatic switch 

Count of responses indicating aircraft that automatically switched to their 

back up C2 link on failure of the primary link. 

106 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 15: Examples of Topics Discussed in CNS Interviews with 

Flight Crews 

Examples of quotes from debriefs with test site crews after a day of flying CNS tests and the 

corresponding topic category. 

CNS 1 

Information: 

Availability 

18 I was watching on one USS, we had multiple USSs today executing

different lost-link things. I couldn’t tell when that happened at all, and I 

was kind of isolated, looking at when XXX’s stuff would go lost link, I’d 

have to listen on the radio to figure out when that even happened. So,

there’s no information provided right now across USSs and, if we were

operating in adjacent air spaces, it would be very valuable I think to share 

in some information. 

Information meets 

user: Information 

desired or wanted 

16 It should also broadcast the last known heading, altitude, and airspeed so

you can take a guess that it is going in that direction and that is a place to 

avoid. 

Operator

experience:

Situation Awareness 

12 You really only have to know – even if someone else is even having 

trouble, you shouldn’t have to know about it unless it’s putting them in

danger or affecting your airspace or where you’re operating. 

Method: Procedures 

for flight/event 

5 If you still have one link and are in control of your vehicle, and you’re

going to stay in your own space, I don’t care, really. Someone cares, but

it’s not me if I’m trying to operation my own mission and you’re not 

coming into my space. 

Automation: Design

of system 

20 We were able to have a flight window open, execute the flight, and be

midway through the flight before the XX USS figured out that is what we

are doing. And then the XX USS closed ours and we had no indication for

why it closed. (This was during the first flight of CNS3 three minutes into 

the flight). 

Concept: Privacy/

sharing 

8 I’m comfortable with the notion of, if you lose one link, and your 

secondary link doesn’t impair your mission at all, doesn’t put any other

aircraft in harm’s way, and you’re confident in the robust, that it’s not

going to impact anybody, I don’t know if you have to share that. 

CNS 2 

Information: 

Reliability/ accuracy 

of information 

11 ... but the fact that the degraded GPS is difficult to get, should give him

some, uh, mean doesn't provide certainty, but it should certainly give

[NASA] some additional comfort in [the] CNS work that it's...not that you

can't jam or anything, but GPS is a highly available and, and uh, is very 

reliable, you know, so much so that even putting a shroud around the

antenna was not creating difficulty. 

Information meets 

user: Accuracy of 

information 

2 It's also important for a troubleshooting perspective to see if the number of 

satellites and the reported HDOP are incongruent. Like if I have high

HDOP but I have 16 satellites, there may be something else going on. 

Operator

experience:

Situation Awareness 

2 So in the event that I lose GPS, I have a general idea of where the aircraft 

is. Even though I may, it may take me a second to figure out what 

orientation I'm in, uh, and I can take over the flight by a manual control and 

return it back to my position or if I feel that I am not comfortable with that,

then I can actually have it lands where it is. 

Method: Procedures 

for flight/event 

4 ... procedures that, that sometimes it's not clear why things are working

and why things aren't and we just have to, you know, reset them a number

of times and, and everything... And eventually, eventually got it. Um, but 

you know, we're, we're learning ... 

107 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Automation: 

Degradation 

10 The biggest problem/fear is RF interferences that changes the flight pattern

of the aircraft, could be GPS signal interference, C2 band interference,

backup RC band interference, or overload on the onboard RF circuits. 

CNS 3 

Information: 

Usefulness of 

information 

7 But the thing is that the interface doesn’t tell us why. It actually grays that 

button out with the alert icon in it. If it closes and tells us specifically why 

it closed you because of time conflict or altitude conflict, otherwise we are

taking another shot in the dark. 

Information meets 

user: Confidence in 

information 

1 That’s probably my biggest concern of stuff we could be flying around – 

power lines or large metal structures (inaudible) interference and you start

kind of seeing it dance around up there. That makes me more nervous than

if my video goes out or something like that. 

Operator

experience: Lack of 

awareness 

1 But for us we didn’t have any explanation in the USS interface for why it

got closed. They had to go on the back end and figure out why. 

Method: Procedures 

for flight/event 

4 So I was piloting the aircraft and basically just looking, watching the

aircraft, I wasn’t paying a lot of attention to the ground control station. I 

had XX watching that for me and just kinda trusted him to tell me if

anything was out of the ordinary or if everything was working like it was

supposed to. I mean, it works fine for us because I can just keep my eyes 

on the aircraft and do it that way. 

Automation: 

Functions 

12 So, from the moment we send time and receive acknowledgment that loop 

should be less than 3 seconds. If it exceeds that then there is something 

wrong there. 

Concept: Design of 

system 

2 Another USS was on our LUN, .... Multiple USS’s can be on that, but, we

just saw another USS that came onto our LUN that was pushing our band

data, and it turned out it was rejecting your plan. 

108 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 16: Examples of Notes from CNS Observations 

A selection of excerpts from NASA representatives’ field observation notes and the corresponding 

topic category. 

CNS 1 

Information: 

Quality of alerts 

There was a visual indicator located on the bottom of the GCS screen, which remained clear 

when all was good but turned red in case of a loss of communication link, notifying the 

operator about the loss. 

Automation: 

Design of system 

and functions 

The link/signal status is monitored with a dedicated application, developed basically for the

specific architecture of the UAS they are using. They are logging and monitoring the network 

and links status over time with a very basic terminal application. At the moment the interface 

is used to check the status only of LTE and Wi-Fi signals since the SATCOM is still not

integrated. The application is test-oriented and color-coded areas are used to check if the 

signal is drop under a specific threshold. For the Wi-Fi link they are also using a dedicated 

commercial application to monitor the status and strength of the signal. 

Information 

required/desired 

The UTM/USS information are monitored as feedback to check if the positions of the UAV

are updated correctly during the signal drop, during the switchover of the communication 

links. The big concern regards the possibility to go non-conforming/rogue if the 

communication switchover takes too long. 

Automation: 

Design of system 

and functions 

The RF datasets have been collected but at the moment such information will be mainly used

to characterize the environment around the test site. Currently there are no plans on how to

integrate such information directly within UTM environment (or if and how real time

evaluations of RF interference can be shared for operations). 

Automation: 

Design of system 

The time to complete the switchover is very short. The adopted solution, which is

architecture-dependent, allows to automate the process of switchover and there is no drop of

information on user/pilot side since the communication links are monitored over time and

kept in a sort of “hot redundancy” to make the transition as seamless as possible. For this

reason, the position updates are checked as normal to be sure that there is no information loss 

because of the process (through the UTM client tool). The tool used for link status monitoring 

is instead checked only to be sure that the automatic switchover behaves as expected. 

Information 

required/desired 

NASA Rep question: If it was your vehicle with the C2 link issue, would you be OK with

sharing that information to other users of your USS? What about neighboring USSs?

TSO Response: Yes. Major issues with communication links can be shared without problems,

also if a level of priority/criticality must be defined/formalized to avoid possible 

communication of false alarms. The communication of C2 link issues can be shared with 

other neighboring USSs once it is communicated to the associated USS. From this point of

view, it is more a matter of data sharing across USSs and not an issue regarding the UAS 

operator, once the C2 problems has been notified. 

Other: 

Operator buy-in 

Once it has been decided to share a subset of the information of a UAS mission (relevant to 

the purposes of traffic control) with other organizations that use UAS for their business there

are no particular constraints for the public. In that scenario the only limit comes from the type

of data that can be exploited/useful to public (to avoid overflow of information). The key 

point is to decide what subset of information is shared outside your own organization. These 

rules apply for any data consumer outside an organization, regardless if they are vehicle 

operators or not.

Inside the same organization there should be no constraints on the amount and type of the

information shared among the vehicle operators. 

Automation: 

Design of system 

LTE was lost because the signal got stuck in a rare feedback loop. 

CNS 3 

Other: 

Confounds 

The GNSS (Global Navigation Satellite System) interference can degrade the position quality

and when it occurs the tracking and positioning signal can be lost completely. Devices like 

chirp jammers can for example disrupt the GPS/GNSS signals quite easily (if not properly 

109 
 

 

 

 

 

 

 

mitigated) and UAV navigation capabilities could be strongly affected by such

technologies/equipment.

Other sources of interference that can result in a reduction or loss of UAV’s ability to

navigate as planned could be represented by some form of self-interference due to payload or 

UAV electronics equipment. 

Information 

meets user: 

Effectiveness of 

data 

There was confusion from the team due to non-conforming. Position updates had a delay for

more than 2 seconds. 

110