NASA/TM-20205006337

Strategic Deconfliction Performance

Results and Analysis from the NASA UTM Technical

Capability Level 4 Demonstration

Joseph L. Rios

Jeffrey Homola

NASA Ames Research Center, Moffett Field, California

Nicholas Craven

Millennium Engineering and Integration Company, Moffett Field, California

Punam Verma

Universities Space Research Association, Moffett Field, California

Vijay Baskaran

SGT Inc., Moffett Field, California

August 2020

1
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

NASA STI Program ... in Profile

Since its founding, NASA has been dedicated

to the advancement of aeronautics and space

science. The NASA scientific and technical

information (STI) program plays a key part in

helping NASA maintain this important role.

The NASA STI program operates under the

auspices of the Agency Chief Information Officer.

It collects, organizes, provides for archiving, and

disseminates NASA’s STI. The NASA STI

program provides access to the NTRS Registered

and its public interface, the NASA Technical

Reports Server, thus providing one of the largest

collections of aeronautical and space science STI

in the world. Results are published in both

non-NASA channels and by NASA in the NASA

STI Report Series, which includes the following

report types:

TECHNICAL PUBLICATION. Reports of

completed research or a major significant

phase of research that present the results of

NASA Programs and include extensive data

or theoretical analysis. Includes compilations of significant scientific and technical

data and information deemed to be of

continuing reference value. NASA

counter-part of peer-reviewed formal

professional papers but has less stringent

limitations on manuscript length and extent of

graphic presentations.

TECHNICAL MEMORANDUM.

Scientific and technical findings that are

preliminary or of specialized interest,

e.g., quick release reports, working

papers, and bibliographies that contain

minimal annotation. Does not contain

extensive analysis.

CONTRACTOR REPORT. Scientific and

technical findings by NASA-sponsored

contractors and grantees.

CONFERENCE PUBLICATION.

Collected papers from scientific and

technical conferences, symposia, seminars,

or other meetings sponsored or

co-sponsored by NASA.

SPECIAL PUBLICATION. Scientific,

technical, or historical information from

NASA programs, projects, and missions,

often concerned with subjects having

substantial public interest.

TECHNICAL TRANSLATION.

English-language translations of foreign

scientific and technical material pertinent to

NASA’s mission.

Specialized services also include organizing

and publishing research results, distributing

specialized research announcements and

feeds, providing information desk and personal

search support, and enabling data exchange

services.

For more information about the NASA STI

program, see the following:

Access the NASA STI program home page

at http://www.sti.nasa.gov

E-mail your question to help@sti.nasa.gov

Phone the NASA STI Information Desk at

757-864-9658

Write to:

NASA STI Information Desk

Mail Stop 148

NASA Langley Research Center

Hampton, VA 23681-2199

2
 

 

 

 

 

NASA/TM-20205006337

Strategic Deconfliction Performance

Results and Analysis from the NASA UTM Technical

Capability Level 4 Demonstration

Joseph L. Rios

Jeffrey Homola

NASA Ames Research Center, Moffett Field, California

Nicholas Craven

Millennium Engineering and Integration Company, Moffett Field, California

Punam Verma

Universities Space Research Association, Moffett Field, California

Vijay Baskaran

SGT Inc., Moffett Field, California

National Aeronautics and Space Administration

Ames Research Center

Moffett Field, California 94035-1000

August 2020

3
This report is available in electronic form at

https://www.sti.nasa.gov

4
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Overview

Unmanned Aircraft System (UAS) Traffic Management (UTM) refers to the

service-based, cooperative approach to the management of small UAS in the National Airspace

System that is safe, scalable, and fair. UTM provides the means to manage the airspace in a

complementary manner that does not burden the current air traffic control workforce or

infrastructure but allows the Air Navigation Service Provider to maintain its regulatory and

operational authority of the airspace.

A key feature of UTM is the ability to provide operators the means to strategically deconflict

operations from others in the airspace through the digital exchange of information via supporting

services. Through this approach, the four-dimensional operation volumes that encompass the

intent of operators in a given area are discoverable and can be used for airspace awareness as

well as planning conflict free operations that account for and avoid other operations. In certain

cases, it is also possible to negotiate volume intersections for shared airspace use without the

need to re-plan.

In the NASA UTM concept, strategic deconfliction is the first layer of three in the overall

conflict management model. The three layers of the conflict management model, which follow

the International Civil Aviation Organization’s scheme [ICAO 2005] are: strategic conflict

management, separate provision, and collision avoidance. In UTM, the strategic layer mostly

occurs prior to departure, but is applicable to en route operations with sufficient planning

horizon. The initial requirements for a strategic deconfliction capability within UTM are defined

in a NASA publication [Rios 2018].

Within the concept and implementation of service-provided strategic deconfliction is the

notion of priority. It is understood that there are instances in which an operation requires a

priority designation within the UTM system and special handling accordingly to provide situation

awareness and facilitate appropriate responses from other airspace users. Examples of

situations requiring priority designation include: when an operator declares an emergency due

to problems with the vehicle or its immediate surroundings; operations that are in support of

certain organizations (e.g., public safety and first responders); or special missions that also

require priority use of airspace (e.g., emergency medical deliveries).

The UTM concept also allows for dynamic constraints in the system. These also relate to

the topic of priority in the sense that the airspace that the constraint encompasses may have

different characteristics. These characteristics of a constraint may indicate operations that are

allowed within the constraint in addition to those which must avoid it through strategic

deconfliction with the volume. Operations that are specially permitted to access the UVR area

are typically assigned priority status given the nature of their mission and their associated

credentials.

The ability to perform strategic deconfliction, handle certain operations with a priority

distinction, and establish constraints that are communicated throughout the UTM system, is

predicated on an architecture that has been established through an evolutionary process in

response to close collaboration with stakeholders from government and industry. Another

important and influential aspect of these capabilities and architecture is the live, distributed flight

tests that have been conducted across the Technical Capability Levels (TCLs) that culminated

with a set of complex tests performed as part of TCL4 [Rios 2020]. The TCL4 flight test involved

two FAA-designated UAS test sites building teams to collaborate with NASA’s UTM Project on

the execution of several detailed, small UAS scenarios in urban environments.

5
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

UTM Operational Architecture

The proposed system for managing small UAS at low altitude is much different than

traditional air traffic management systems for commercial aircraft in the NAS. Within UTM, the

operations are managed collaboratively by a collection of UAS Service Suppliers (USS). Each

small UAS operation is managed by a USS to provide appropriate operational data to other

USSs and to the operator. These data exchanges are in support of the Concept of Operations

for UTM [Kopardekar 2016][FAA 2018][FAA 2020] and defined by a set of Application

Programming Interfaces and UAS Service Supplier Specification [Rios 2019b]. The Air

Navigation Service Provider (ANSP) still maintains authority over the airspace, but certain

services are delegated to the USSs so that the ANSP does not directly manage this class of

operations in nominal cases. These services include strategic deconfliction of operations. The

high-level architecture is provided in Figure 1.

Figure 1: UTM high-level architecture.

TCL4 Flight Test Summary

The overall planning and execution of TCL4 is described in a standalone publication [Rios

2020]. The following brief summary is intended to give context for the rest of the document.

The TCL4 Demonstration was the final in a series of demonstrations by the NASA UTM

Project. Each demonstration from TCL1 to TCL4 increased in the number of capabilities and

the complexity of the operational environment. Overall, the UTM Project sought to demonstrate,

in concert with the FAA and industry, how UTM should work and to identify areas in need of

further research. UTM is such a paradigm shift in how the airspace will be managed that often

the anticipated challenges are not easily known. The TCL Demonstrations helped shape the

6
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

architecture and concept of UTM and aided in the identification of gaps and formulation of

requirements.

TCL4 was executed at two of the FAA-designated UAS Test Sites: Nevada and Texas,

managed by the Nevada Institute for Autonomous Systems and Lone Star UAS Center of

Excellence & Innovation, respectively. Both sites were provided the same statement of work

(SOW) and each developed a series of test plans to meet the SOW. These plans were finalized

in coordination with the NASA UTM Project.

The test sites developed a team of partners to execute the flight testing. These included

USS providers, UAS equipment manufacturers, weather service providers, cellular service

providers, radar providers, operators, public safety agencies, and others. The SOW described

five detailed scenarios with specific characteristics and test events that were considered the

primary requirements of the flight test. The scenarios were run multiple times in TCL4 and each

run involved live and simulated aircraft to safely create the desired interactions and generate the

necessary density to test the system and concepts. The five scenarios are briefly presented in

Table 1.

Table 1. Summary of scenarios in TCL4.

Scenario Description

1 Nominal high density UTM operations with multiple mission types. A sudden, simulated weather

event is forecast to impact the area, which results in a UVR. Operators with current or planned

operations within the UVR geometry respond by returning to the launch point, re-planning to

utilize an identified safe landing location that is de-conflicted from other operations, or avoiding

takeoff from within the UVR.

2 A pop-up concert takes place at a local venue that results in a number of UTM operations in the

area. An public safety incident occurs at the concert resulting in a UVR being established to allow

for a Public Safety UTM response and clearance of non-essential operations from the area. Those

operations clearing the UVR must replan and de-conflict to ensure safe exit. UTM-enabled Remote

ID is used to identify and contact operators that have not cleared the area. Special access to the

UVR is granted for news coverage in addition to the public safety vehicles.

3 UTM operations are being conducted near an active airport with a medium density of sUAS

operations. Reported events take place that require Remote ID of specific operations as well as

security responses to monitor situations near the airport. Piloted general aviation aircraft are

conducting flights in the area and one aircraft’s path results in a conflict with a UTM operation’s

operation volume with a subsequent response from the UAS operator.

4 A simulated low battery situation forces a vehicle to land quickly, which requires nearby

operations to re-plan and avoid the landing vehicle. Later, large-scale loss of communication and

navigation events are experienced that require contingency management procedures in response.

Remote ID is requested to identify vehicles that are reported to be congregating in a particular

area during the events.

5 Multiple events are taking place in a suburban area, which draws a gradually increasing number of

sUAS performing a variety of supporting missions. The increase in density is accommodated

through strategic deconfliction that eventually requires UAS Service Supplier (USS) negotiations

for re-planning or agreement to allow overlapping Operation Volumes. Operational density in one

area increases with subsequent negotiations that result in shared airspace for multiple vehicles

and a transition to cooperative separation. One USS supporting multiple operations experiences a

critical failure, which results in contingency management procedures for affected flights and a

switch to an alternate USS where able.

7
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

USS implementers were a key set of partners. This group represents key stakeholders in

the future of the UTM System. Given the primacy of the USSs in UTM, NASA developed a

series of collaborative simulations and checkout exercises that each USS needed to complete in

order to be allowed to participate in the TCL4 flight testing. Some of this process is described in

[Rios 2018] and [Rios 2018b]. This process also helped NASA inform potential future

requirements related to USS Checkout procedures for a future operational UTM System [Smith

2019].

In the months leading up to the flight testing, the test sites worked with NASA to execute

tabletop exercises, simulations, and shakeout flights to prepare for the actual field tests. At

each of these stages, the data collection approach was solidified to support NASA’s analysis

requirements.

Measures of Performance

The NASA UTM Project defined twenty Measures of Performance (MOPs) to aid in

quantifying the UTM System as designed and tested in TCL4. Each of these MOPs will be

reported internally to NASA. Additionally, NASA will report out on most of these MOPs in

various venues. Three of the MOPs will be detailed in this report.

The approach to defining MOPs is driven by NASA’s “NASA Systems Engineering

Processes and Requirements” [NASA 2013]. Requirements for a system are supported by

Measures of Effectiveness (MOEs) which are qualitative in nature. These MOEs are supported

by MOPs which are quantitative in nature. Formal definitions for requirements, MOEs, and

MOPs are provided below. These relationships are illustrated in Figure 2.

Figure 2. Relationships of requirements, MOEs, and MOPs.

The UTM System has several high-level requirements. The formal definition of a

requirement used within the UTM Project is defined as:

The agreed upon need, capability, capacity, or demand for personnel, equipment,

facilities, or other resources or services by specified quantities for specific periods of

time or at a specified time expressed as a "shall" statement. Acceptable form for a

requirement statement is individually clear, correct, feasible to obtain, unambiguous in

meaning, and can be validated at the level of the system structure at which stated. In

pairs of requirement statements or as a set, collectively, they are not redundant, are

adequately related with respect to terms used, and are not in conflict with one another.

Each MOP is connected to a defined Measure of Effectiveness (MOE). An MOE is defined

by NASA as:

A measure by which a stakeholder's expectations will be judged in assessing

satisfaction with products or systems produced and delivered in accordance with the

associated technical effort. An MOE is deemed to be critical to not only the acceptability

of the product by the stakeholder but also critical to operational/mission usage. An MOE

8
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

is typically qualitative in nature or not able to be used directly as a "design-to"

requirement.

A MOP is defined by NASA [NASA 2013] as:

A quantitative measure that, when met by the design solution, will help ensure that

an MOE for a product or system will be satisfied. MOPs are given special attention

during design to ensure that the MOEs with which they are associated are met. There

are generally two or more measures of performance for each MOE.

The requirements supported by the MOPs in this document are listed in Table 2.

Table 2. Summary of requirements supported by MOPs in this document.

REQ ID Title Description Source

UTM-REQ-1 The UTM System SHALL aid

in sUAS staying clear of each

other.

A fundamental feature of an air traffic

management system is to aid in

keeping aircraft safe, including from

each other.

[NASA 2016]

UTM-REQ-2 The UTM System SHALL aid

in sUAS staying clear of

traditional aviation.

Traditional aviation encompasses

many forms of air vehicles and

operation types. UTM needs to aid

sUAS operators in staying clear of

these operations.

[NASA 2016]

UTM-REQ-5 The UTM System SHALL

allow for priority of Public

Safety operations over other

nominal operations.

Public safety operations need to

perform their essential tasks safely,

without interference from nominal

operations. UTM protocols allow for

this need.

[NASA 2016]

Summary of Measures of Performance

Table 3 provides a summary of the three MOPs reported in this document. Further details

are provided in the subsequent subsections.

9
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3. Summary of MOPs reported in this document.

MOP ID Title Description Minimum

Success

Target

Success

UTM-MOP-01 Strategic

deconfliction rate

Percentage of nominal (i.e.

non-emergency, non-priority) ACTIVE

UTM Operations that have 4D disjoint

Operation Plans OR have a Negotiation

Agreement to allow intersection with

each other Operation that intersects

with it.

> 90% 100%

UTM-MOP-18 Replan rate in

response to priority

operation

Calculate the percentage of operations

affected by a pop-up priority operation

that are able to replan to deconflict

with that priority operation prior to

departure of the pop-up priority

operation.

> 90% 100%

UTM-MOP-20 Deconfliction rate for

priority operations

Percentage of public safety operations

with a given lead time that have

strategically deconflicted plans, i.e,

public safety operations that have no

conflicts (prior to departure) with

existing operations.

95% @ 5

min lead

100% @ 5

min lead

NASA UTM Data Collection Architecture

The UTM data collection architecture is the result of lessons learned from previous TCLs as

well as industry best-practices as applied to UTM testing. A key requirement for partner

participation with NASA in UTM flight testing was the delivery of data for further analysis. There

are two primary flows of data into NASA from the stakeholders involved in testing. First, there is

the data exchanged as part of the operational concept. These would include data such as

operation plans, positions, and other UTM messages. All of these data were collected live with

a designated USS Data Collector (UDC) that interacts with the other USSs as if it were a USS.

The other flow of data was specified as part of NASA’s Data Management Plan for TCL4.

These include data that are used for research purposes and are not necessarily part of the UTM

operational concept. These data would include elements such as latency measurements,

vehicle logs, and weather data. These data were submitted to a Data Management Plan (DMP)

system during or after testing.

Both flows of data end up in a Universal Data Store (UDS). The UDS within this architecture

provides a single source of truth for all TCL4 data, implementing a key architectural pattern in

data collection and analysis. UDS is used for all visualization and analysis of TCL4 testing data.

Figure 3 summarizes this data collection architecture.

10
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 3. TCL4 data management architecture.

General Data Reduction

All data were collected during each run of each scenario. However, not all data in the TCL4

datastore were intended or appropriate for calculation of each MOP. Thus, several layers of

filtering are required to ensure that the appropriate data are used as starting points for the

analysis. In general, live and simulated flights were treated the same in terms of data collection.

Each specific operation had data requirements levied upon it depending on several variables,

including for example whether it was live vs. simulation, whether it was part of a particular

scenario, or if it was a particular vehicle type. Some data collection was limited to live vehicles

with, for example, certain equipment installed that would not have been reasonable to collect

from a simulated aircraft. Some analysis was applicable only to live aircraft, and is noted as

such.

To determine valid operations for analysis, the metadata of each model is filtered as follows:

1. Only models identified by the submitter as valid via a metadata flag

2. Only operations that changed altitude by 75ft, which is based on the standard gps error

model of ~2 sigma.

3. Only valid demonstration scenarios, with corresponding valid run numbers. Some

scenarios are for testing or shakeout of the operations or various subsystems and are

not intended for data analysis.

11
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

After flight testing was complete, the NASA team determined that 23 total runs were of

sufficient quality (had sufficient duration, with appropriate threshold of active operations, and all

core systems functioning normally) for analysis of strategic deconfliction MOPs (i.e. those

presented in this paper). The result of the general data reduction is illustrated in Figure 4.

Further filtering per the needs of each MOP in this paper will be described in those sections.

Figure 4. Filtering of operations with appropriate characteristics for analysis.

Measure of Performance 01: Strategic Deconfliction Rate

UTM-MOP-01 is titled “Strategic deconfliction rate.” The high-level description is

“Percentage of nominal (i.e. non-emergency, non-priority) ACTIVE UTM Operations that have

4D-disjoint Operation Plans OR have a Negotiation Agreement to allow intersection with each

other Operation that intersects with it.” An “intersection” occurs when at least one 4D volume of

one operations intersects with at least one 4D volume of another operation. Given previous

experience in flight testing the UTM System, the success criterion was set at 95% with a target

success of 100%.

This MOP is fundamental to the UTM concept. UTM is designed to provide a mechanism for

small UAS operators to share information about their operations with other stakeholders,

including other operators. The MOP supports UTM-MOE-1 by ensuring that operations

endeavor to stay clear of each other on a strategic time horizon (typically prior to flight, but

sometimes due to changes while in flight), via the use of their respective USSs.

Data Reduction

For this analysis, the valid operations identified in the ‘General Data Reduction’ section were

used as the preliminary filter. Coordinating a flight test of multiple live vehicles and multiple

simulated vehicles in an urban environment is a fragile endeavor. Weather, networking,

subsystems, crew readiness, and many other issues can delay or halt the run of a given

scenario. After flight testing was complete, the NASA team determined that 23 total runs were

of sufficient quality for analysis of strategic deconfliction MOPs. Thus, all operations from other

runs were excluded from analysis.

Since UTM-MOP-01 is for “nominal” operations, all operations that indicated an off-nominal

state either via its operation plans or UTM messages were also excluded. Those operations are

explored in other MOPs. In addition, some operations were submitted to the UTM System, but

12
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

never flew. So, if there were no position reports indicating a change in altitude of at least 75

feet, an operation was excluded from analysis.

Ultimately, this filtering process left 327 operations from the 23 quality runs (as defined

above) across both test sites (Texas and Nevada) as shown in Figure 5, with good

representation of live and simulated operations.

Figure 5. Breakdown of the 327 operations used for UTM-MOPS-01 analysis.

Results

From the 327 operations under analysis for UTM-MOP-01, there were several intersecting

operations. However, every intersection is not inherently “bad.” The UTM concept allows for

intersecting operations if the operators have an agreement to do so. This “negotiation

agreement” is the outcome of a series of message exchanges. When operators agree to let

their operation plans intersect, they are indicating that separation during the time of intersecting

operation volumes is assured via some other means, likely involving technologies on-board the

vehicle. Operations that have a negotiation agreement codify that agreement by updating their

operation plans indicating to each other and other stakeholders that the intersection is agreed

upon. Such operations are considered to be strategically deconflicted.

There were 208 pairs of operations with intersections AND corresponding negotiation

agreements. There were 185 distinct operations involved with those 208 operation pairs.

There were 8 operation pairs, across 16 distinct operations, that were in conflict, because

they were not strategically deconflicted. All of these conflicting operations were simulations; no

live operations were in conflict with any other operation. With 16 out of the 327 operations in

conflict, the success criterion of 95% for UTM-MOP-01 was met.

The most interesting aspect of the failure to deconflict is that nearly all of them involved pairs

of operations that were managed by the same USS. Only 1 of the 8 conflict pairs involved

operations managed by different USSs. This is illustrated in Figure 6. Further discussion of this

finding is provided in the Discussion section below.

13
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 6. Most unresolved conflicts occurred between two operations managed by the same USS.

Discussion

There are three key findings from the data collection and analysis related to UTM-MOP-01.

This section will list and provide discussion on each.

Strategic Deconfliction is Effective

USSs and their operators effectively used the strategic deconfliction concept and its

implementation within TCL4. Over time within the UTM concept developed at NASA, strategic

deconfliction evolved to allow negotiations and those negotiations allowed for operators to agree

to intersect under certain conditions. Generally, operations were successfully deconflicted prior

to departure, even in relatively dense scenarios.

Intra-USS Negotiations

The majority of the conflicts that were not captured in TCL4 involved operations managed by

the same USS. This implies an issue with the approach to managing conflicts internal to a

given USS. Without access to the algorithms implemented by the independent USS

developers, it is difficult to know why operations within a given USS might not be deconflicted.

In the NASA USS implementation, a simple approach was taken. The NASA USS would reject

an operator’s plan if it intersected with an existing plan managed by the NASA USS, thus

ensuring that the NASA USS would not support two operations that overlapped. This would

trigger that rejected operator to replan to avoid the existing operation. If an operator submitted

a plan that did not intersect an existing plan managed by the NASA USS, but did intersect a

plan managed by an external USS, the negotiation process would be initiated automatically by

the NASA USS with that external USS.

It appeared that some USSs accepted all operations from the operators they supported, with

the intention of deconflicting them via negotiation with external USSs. Without pilot debriefs of

every one of the 100’s of operations correlated against the data, it is unclear why a USS would

allow intersections within operations it is managing. As a hypothesis based on working on USS

designs and the partner systems, some partner USS processes apparently simply neglected

checking against operations they were already managing. This indicates the need for explicit

requirements on future operational USSs that they indeed deconflict their own operations with

the same due diligence they use with external USSs. This can be as simple as the USS

14
 

 

 

 

 

 

 

 

 

 

 

 

ensuring that intersecting operations that it manages establish a negotiation agreement to

intersect. This agreement would be visible to other stakeholders to indicate that the intersection

is known and is being managed by the respective operators and the USS.

Protocol Adherence

Negotiations are a potentially complex activity with several edge cases. Due to the needs of

the flight test, lack of fully developed protocols, and relatively short development time, many

USSs implemented “shortcuts” in negotiation. These might take the form of internal rules that

stated things like “accept all intersection requests from simulated aircraft” or “reject all

negotiations for live operations.” These are valid approaches in terms of the negotiation

protocols for TCL4 in that they did not break any established rules. The logic required to more

robustly implement negotiation would take significant effort and a well-defined specification.

This is an important consideration for future standards development and future checkout

processes for USSs.

15
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Measures of Performance 18 and 20: Pop-up Priority Operations

UTM-MOP-18 is entitled “Replan rate in response to priority operation.” The high-level

description is documented as “Calculate the percentage of operations affected by a pop-up

priority operation that are able to replan to deconflict with that priority operation prior to

departure of the pop-up priority operation.” The minimum success criteria for MOP-18 was

documented at greater than 90% with a target success of 100%. UTM-MOP-20 is titled

“Deconfliction rate for priority operations.” The high-level description is documented as

“Calculate the percentage of public safety operations with a given lead time that have

strategically deconflicted plans, i.e, public safety operations that have no conflicts (prior to

departure) with existing operations.” The minimum success criteria for MOP-20 was

documented using a 5-minute lookahead time at 95% with a target success of 100%.

Due to the tempo of operations and the need to keep the orchestration of flights progressing,

the priority operations did not typically depart later than 5 minutes after the operation was

announced via the USS Network. Thus, the lookahead requirement was dropped for analysis.

In addition, operators deconflicted from the pop-up priority operation in various ways that were

not considered in the drafting of the MOP. This may have been in response to the fact that

there was insufficient time from announcement of the priority operation to that operation

entering the airspace, or a lack of understanding of the scenario or concept by stakeholders, or

other reasons. Given the execution anomalies, these MOPs as originally documented are

officially cataloged as “inconclusive” for TCL4. However, there are many insights to be gained

through examination of the collected data. This section examines the conflicts that the priority

operations and nominal operations faced, regardless of lookahead time as well as the varied

approaches that operators took to minimize those conflicts.

Data

Operations within the UTM concept as tested in TCL4 would announce their priority status

as part of their operation plan using specific data elements. Though, in certain scenarios, an

operation may be perceived as having some “priority” due to its access to otherwise closed

resources.

Specifically, an operation may be added to a whitelist of operations that are allowed into a

newly announced UAS Volume Reservation (UVR). A UVR is a constraint in the UTM airspace

that is typically short-lived and dynamically announced. UVRs support use cases such as

pop-up events on the ground, extreme weather events, public safety events (airborne or

ground-based), etc. By default all operations are excluded from a UVR. However, if an

operation has its identifier added to the UVR as a permitted operation, it would be permitted to

have plans that intersect the UVR and could therefore enter it. This analysis includes such

operations as “priority” operations to gain insight into how they interacted with other operations.

From the complete list of operations available for this MOPs analysis (652, as outlined in the

General Data Reduction section), there were 21 operations that qualified as priority operations,

either by announcing their status in the operation plan or by inclusion within a UVR

announcement.

These priority operations in TCL4 would often (by design) intersect existing nominal

operations (both simulated and live). The operation plan models and their respective operation

volumes provided the 4D information to calculate any intersections in plans, allowing insight into

whether active, nominal operations could remove intersections between their active plans and

the newly announced priority operation. It is important to note in this testing that the exact

moment of a priority operation being announced was not always known by participants, but they

16
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

did know (via scenario design) that such a pop-up operation would occur. As always, range

safety protocols kept operations safely separated regardless of the state of the UTM systems

under test.

There were 64 nominal operations that intersected with the 21 priority operations, upon

announcement of the priority operation (either via UVR or the operation itself), resulting in 85

total operations for analysis. Of those 64 nominal operations intersecting with the priority

operations upon announcement, 15 remained in conflict upon the priority operation activating its

plan. Those 15 operations intersected with 5 distinct priority operations, thus some priority

operations were intersecting with multiple nominal operations upon take off. Of the 5 distinct

priority operations. The actions of the 15 nominal and 5 priority intersecting operations are

analyzed in the next section along with overall analysis for these MOPs.

Including the operations that were whitelisted in a UVR as a “priority” operation greatly

expands the documented definitions of priority operations within UTM. The operators flying

these missions may have had an implicit notion that they were flying a “priority” operation due to

exemption from an announced UVR. However, to require existing operations to deconflict from

a newly announced operation, that newly announced operation would need to include the

appropriate priority data elements in its operation plan. In the UTM concept, if these elements

are not included in the operation plan, the operation is still considered nominal, and it has an

added ability to fly into a UVR. However, by including these implicitly priority operations,

additional data are available that may provide insights into the use of the UTM system in TCL4.

Analysis

As mentioned above, there were 15 nominal operations still in conflict with priority

operations upon takeoff time of the priority operation. These 15 conflicting nominal operations

fell into one of four distinct categories:

1. The nominal operation keeps its aircraft clear (i.e. the position of the operation is not

inside) of the priority operation’s volume, even though its volumes intersect with priority

operation’s volumes

2. The nominal operation becomes an emergency operation itself

3. The nominal operation transits through the active priority operation’s volumes with a

negotiation agreement

4. The nominal operation transits through the active priority operation’s volumes without a

negotiation agreement

These categories and the overall counts are summarized in Figure 7.

17
 

 

 

 

 

 

 

 

 

 

 

 

Figure 7. Summary of the conflicting operations for MOPs 18 and 20.

To further understand how and why operations were not able to maintain strategic

deconfliction, it is illustrative to examine the outlier cases. In the following sections, a closer

look at two of those cases is provided.

Case 1: Misbehaved Priority Operation

In Figure 11, there is a nominal operation that failed to deconflict with a pop-up priority

operation and transits the priority operation’s volumes without a negotiation agreement. Upon

first reading of such a statement, it would be natural to assume that the affected nominal

operation failed in its UTM duties to deconflict from a priority operation and further endangered

the airspace and surrounding areas by transiting that priority operation’s volumes. However, a

closer look at the timeline of events sheds light on the actual culprit in this interaction. The

timeline is depicted in Figure 8.

18
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 8. Timeline of events for simulated operation cc5c interacting with operation 5b04.

As noted above, the planned activity in this scenario was for a UVR to be announced with at

least a five-minute lead time from announcement to activation. In that announcement, certain

operations would be “whitelisted” via their flight identifiers and allowed to continue operation

within the UVR or change their plans to enter the impending UVR. The whitelisted operation is

not a priority operation in the technical definitions within UTM, but is allowed to enter the UVR.

Operation 5b04 is the whitelisted operation, with operation cc5c being an airborne nominal

operation that is not whitelisted. In this scenario, cc5c is simulated and 5b04 is a real operation,

but their interactions via their USSs occur as if they are both real operations. Operation 5b04

submitted its original operation plan to the TCL4 data collection system with volumes

intersecting the existing cc5c operation. This is not allowed per UTM protocols, as each

operation needs to be deconflicted from every other existing operation or have a negotiation

agreement in place to allow operations to intersect. Operation 5b04 did have an negotiation

agreement with two other operations, but not with cc5c. This is most likely due to the USS for

cc5c rejecting a negotiation request, which is a valid response to such a request. It appears

that 5b04 used the conflicting plan anyway and began its mission.

After commencing flight, 5b04 and the other operations received information of a UVR via

their USSs . That UVR had a scheduled start time just a couple minutes into the future.

Operation 5b04 was on the whitelist for this UVR. In the scenario, the plan was for operation

5b04 to head to the UVR to assist in a mock public safety activity. 5b04 did go to the UVR as

designed in the scenario, but did not update its plan, thus leaving its planned volumes and

becoming ROGUE per UTM specifications. The supporting USS tracked that status

appropriately and applied the state per the UTM specification. The disconnect in procedures

was between the USS and the operator for 5b04. Figure 9 depicts the two interacting flights

shortly after the announcement of the UVR (denoted by yellow boundary) prior to the descent

initiation of 5b04.

Operation cc5c had no escape from conflict. Much like a shopper in a grocery store happily

shopping and then having the manager turn the lights out, lock the door, and then accuse the

shopper of trespassing: Operation cc5c was caught in a situation where it could not get out of

the airspace without further breaking boundaries. This is why it had to descend through 5b04’s

19
 

 

 

 

operation volumes to land (see Figure 10). Thus, the against-protocol actions of 5b04, had a

cascading effect on another operation, forcing it to break protocols to safely exit the airspace.

Figure 9. 5b04 (blue) and priority cc5c (green) inside UVR (yellow boundary).

Figure 10. 5b04 forced to descend through cc5c’s active volume below to clear UVR.

20
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Case 2: UVR begets Emergency

In another example of how operations dealt with the pop up of the UVR, Operation 8b95 was

another live operation in flight when 5b04 was announced and took flight. A depiction of the

timeline of events between these two operations is shown in Figure 15.

Figure 11. Timeline of events for simulated operation 8b95 interacting with operation 5b04.

In this case 8b95 and 5b04 had an initial negotiation agreement to intersect and rely on

other means of separation (Figure 16). When the UVR occurred, 8b95 did not have time to land

without intersecting the UVR. So, 10 seconds after the UVR was activated, 8b95 declared itself

in an emergency state (within the context of the flight test, not in practical operational terms) and

proceeded to head for a safe landing. The updated plan to reach this landing included another

intersection with 5b04, which they codified with a new negotiation agreement (Figure 17).

This would be considered a special case of how deconfliction should work in UTM, but it was

within protocol. It illustrates the effective use of the protocols in a time-constrained scenario,

with some decisions likely made in real-time. The system also clearly supported the ability of

the flight crews and the USSs to handle these negotiations.

21
 

 

 

 

 

Figure 12. 8b95 (blue) and 5b04 (green) with negotiated intersect prior to UVR.

Figure 13. In response to UVR, 8b95 declared emergency and updated operation to

return and land safely with negotiated intersection with 5b04.

Discussion

There are several key takeaway messages from the data collection and analysis related to

UTM-MOP-18 and UTM-MOP-20. This section will list and provide discussion on each.

22
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

UVR Warning Time

Without appropriate time between the announcement and activation of a UVR, a compliant

operation may be forced into non-compliance to safely stay clear of the UVR. This was

understood prior to TCL4 [Homola, 2019], but due to the acceleration of certain testing

conditions, the practical effects of a limited warning time were observed.

Operator Understanding

The actions of certain operations in the context of priority announcements and UVR

implementations indicate a lack of understanding of the UTM protocols. This has been an

understood deficiency for some time in the context of all TCL activities over the years.

Operators were better informed in TCL4 than in prior tests, but the protocols and actions were

more complex than before. These priority and UVR scenarios highlighted this gap. Overall this

can be seen as a shortcoming in NASA’s training for the TCL4 test partners. With more time

and resources, operator education would be a valuable addition to future tests. In addition, this

finding highlights the need to ensure all operators and pilots within the future operational UTM

environment receive appropriate training and education on UTM concepts and protocols. There

is more to operating in this collaborative environment than flying an individual mission under

current day rules and procedures. For further discussion on operator feedback on the UTM

TCL4 effort, see [Martin 2020a] for Texas operation feedback and [Martin 2020b] for Nevada

operator feedback.

Restriction breeds Creativity

Even with the acceleration of TCL4 flight schedules and the difficulty in finding appropriate

deconfliction solutions, there is evidence that many operators and USSs were able to find viable

approaches to constrained situations and stay within the UTM protocols. This is an important

highlight, especially in light of the limited training provided to stakeholders on UTM concepts as

noted above. It is encouraging to note that the concept as tested in TCL4 can support data

exchanges that lead to safer operations in unanticipated scenarios.

23
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Overall Discussion

The analysis of the three MOPs presented in this document each revealed a selection of

insights. Those are summarized in Table 4.

Table 4. Summary of lessons and insights from strategic negotiation testing.

MOP Lesson/Insight Summary

UTM-MOP-01 Strategic Deconfliction

Effective

Overall, USSs were able to share data about each others’

operations under management and use that information to keep

them separated. When separation was not possible or perhaps

inefficient, USSs were able to negotiate intersections where

appropriate.

UTM-MOP-01 Intra-USS Deconfliction USSs are typically checked-out on their ability to deconflict

operations with other USSs. It is also important to ensure that a

USS deconflicts the operations it is managing via targeted

checkout tests for those use cases. Otherwise, there is a distinct

possibility that operations may be in conflict due to USS errors.

UTM-MOP-01 Protocol Adherence Negotiation protocols may be quite complex to handle all use

cases. If overly complex, there is an implementation cost that

may encourage the use of simplified responses to negotiation

procedures.

UTM-MOP-18

UTM-MOP-20

UVR Warning Time As hypothesized prior to TCL4, a UVR that does not offer enough

time for operations to react safely and efficiently, will introduce

complexity and difficulty onto operators and USSs. Appropriate

lead time from UVR announcement to UVR activation is critical.

UTM-MOP-18

UTM-MOP-20

Operator Understanding Some actions during the scenarios indicate a lack of

understanding of the protocols and concepts of UTM on the part

of the operators. In TCL4 this was not the direct fault of the

operators, but it does indicate the need to properly train all

stakeholders for their role in an future operational version of

UTM. In a future operational system, operators are ultimately

responsible for the safety of their operations and knowledge of

the involved systems.

UTM-MOP-18

UTM-MOP-20

Restriction breeds

Creativity

Even when faced with a difficult situation in terms of a sudden

UVR and dense airspace, operators and USSs could find

legitimate, in-specification approaches to handling the situation.

The MOPs for strategic deconfliction provide insight into how a future operational UTM

System may aid in the safe separation of beyond visual-line-of-sight operations of sUAS. With

further efforts on protocol development and standardization, the USS-USS communications

related to strategic deconfliction will be vital to enabling safe access to orders of magnitude

more operations than are possible in today’s environment.

The work of the NASA UTM Project is being transferred to the FAA via a Research

Transition Team. Some aspects are also being transferred to industry via standards bodies and

publications such as this one. NASA plans to continue research on the UTM concept. This

includes graduating concepts and architectures to other aviation domains such as Urban Air

24
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mobility (UAM), high altitude operations (over 60,000 ft), and space traffic management. Thus,

the results from UTM testing will likely have broad impact on future aerospace applications.

References

[ICAO 2005] ICAO, Global Air Traffic Management Operational Concept, Document 9854,

https://www.icao.int/Meetings/anconf12/Document%20Archive/9854_cons_en[1].pdf, 2005.

[FAA 2018] FAA NextGen Office, "Unmanned Aircraft Systems (UAS) Traffic Management

(UTM) Concept of Operations", v1.0, May 2018.

[FAA 2020] FAA NextGen Office, "Unmanned Aircraft Systems (UAS) Traffic Management

(UTM) Concept of Operations", v2.0, March 2020.

[Homola 2019] Homola, J., Martin, L., Cencetti, M., and Aweiss, A., “UAS Traffic Management

(UTM) Technical Capability Level 3 (TCL3) Flight Demonstration: Concept Tests and Results,”

38th Digital Avionics Systems Conference (DASC2019), September 2019.

[Kopardekar 2016] Kopardekar, P., Rios, J., Prevot, T., Johnson, M., Jung, J., Robinson III, J.,

"Unmanned Aircraft System Traffic Management (UTM) Concept of Operations", AIAA Aviation

Forum 2016, June 2016.

[Martin 2020a] Martin, L., Wolter, C., Jobe, K., Goodyear, M., Manzano, M., Cencetti, M.,

Mercer, J., Homola, J., “TCL4 UTM (UAS Traffic Management) Texas 2019 Flight Tests,

Airspace Operations Laboratory (AOL) Report,” NASA TM-2020-220516, March 2020.

[Martin 2020b] Martin, L., Wolter, C., Jobe, K., Manzano, M., Blandin, S., Cencetti, M.,

Claudatos, L., Mercer, J. & Homola, J., “TCL4 UTM (UAS Traffic Management) Nevada 2019

flight tests, Airspace Operations Laboratory (AOL) report,” NASA Technical Memorandum,

submitted for publication.

[NASA 2013] NASA, NASA Systems Engineering Processes and Requirements, NPR 7123.1B,

April 2013.

[Rios 2018] Rios, J., Smith I., Venkatesen P., Smith, D., Baskaran, V., Jurcak, S., Strauss, R.,

Iyer, S., Verma, P., “UTM UAS Service Supplier Development: Sprint 1 Toward Technical

Capability Level 4”, NASA Technical Memorandum, NASA/TM-2018-220024, November 2018.

[Rios 2018b] Rios, J., Smith I., Venkatesen P., Smith, D., Baskaran, V., Jurcak, S., Iyer, S.,

Verma, P., “UTM UAS Service Supplier Development: Sprint 2 Toward Technical Capability

Level 4”, NASA Technical Memorandum, NASA/TM-2018-220050, December 2018.

[Rios 2019] Rios, J., Smith I., Venkatesen P., “UTM Authentication and Authorization

Framework”, NASA Technical Memorandum, NASA/TM-2019-220364, October 2019.

25
 

 

 

 

 

[Rios 2019b] Rios, J., et al., “UAS Service Supplier Specification,” NASA Technical

Memorandum, NASA/TM-2019-220376, October 2019.

[Smith 2019] Smith, I., Rios, J., Mulfinger, D., Baskaran, V., Verma, P., “UAS Service Supplier

Checkout: How UTM Confirmed Readiness of Flight Tests with UAS Service Suppliers,” NASA

TM–2019–220456, December 2019.

26