1 

Toward Developing MTEs for Multirotor

sUAS in Controlled Wind Conditions

Amanda K. Lampton, Ph.D.1 , David H. Klyde2 , Tanner Prince3 , and 

Thomas Swaney4 

Systems Technology, Inc., Hawthorne, CA, 90250, USA 

Christine M. Belcastro, Ph.D.5 

NASA Langley Research Center, Hampton, VA, 23681, USA 

Unmanned aircraft systems are quickly becoming a ubiquitous presence in military and 

civilian life. Developers are demanding they be allowed in the National Airspace System. This 

includes myriad small unmanned aircraft systems that seek to use the uncontrolled airspace 

at altitudes below 500 ft. Methods to verify, validate, and certify these aircraft for safe 

introduction into the airspace, especially for beyond line of sight operations, as well as the 

communication infrastructure to support them are slow to develop for a number of reasons. 

One such is that little to no experimental data of these aircraft performing well defined tasks 

with well defined requirements is available with which to support the development of these 

methods and processes. To this end, a study was conducted to support the effort to develop, 

test, and validate tasks for small unmanned aircraft systems in off nominal conditions. This 

study focused on multirotor aircraft free flying four defined tasks precision hover, lateral 

sidestep, vertical reposition, and landing in controlled wind conditions generated in the 

NASA Langley Research Center 14x22 ft subsonic wind tunnel. Two pilots flew two 

multirotor aircraft through a series of tasks at constant windspeeds ranging from 0 mph to 25 

mph. Performance assessments signaled that the pilots could fly the tasks and meet most 

performance requirements. Increased windspeed resulted in performance degradation as 

expected. Pilot feedback suggested that the course layout can be improved to facilitate 

longitudinal visual cueing. Results indicated that the foundation of these tasks and the process 

by which they were tested in off nominal conditions is sound. 

I. Nomenclature 

H Hover 

L Landing 

LaRC Langley Research Center 

LS Lateral Sidestep 

MTE Mission Task Element 

NASA National Aeronautics and Space Administration 

STI Systems Technology, Inc. 

sUAS small Unmanned Aircraft Systems 

VR Vertical Reposition 

1 Senior Research Engineer, Engineering Services, AIAA Associate Fellow. 2 Vice President and Technical Director, Engineering Services, AIAA Fellow. 

3 Engineering Intern, Engineering Services. 

4 Engineering Intern, Engineering Services. 

5 Distinguished Research Engineer, Dynamic Systems and Control Branch, AIAA Associate Fellow. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 AIAA Scitech 2020 Forum 

 6-10 January 2020, Orlando, FL 

 10.2514/6.2020-1507 

 Copyright © 2020 by Systems Technology, Inc.. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission. 

 AIAA SciTech Forum 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

-

-

-

—

—

-

-

= 

= 

= 

= 

= 

= 

= 

= 

= 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

II. Introduction 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

The presence of small unmanned aircraft systems (sUAS) is quickly becoming ubiquitous in both military and 

civilian life. In particular, a number of civilian applications of sUAS in low-altitude airspace are being developed, 

including cargo delivery, infrastructure monitoring, precision agriculture, search and rescue, etc.1,2 As these programs 

progress and look toward integrating into this airspace, it is necessary to consider the future interactions with current 

users of that airspace (e.g., general aviation aircraft, helicopters, gliders, balloons, parachutists) and how to uphold the 

current level of safety of these operations. 

Current users of this airspace require line of sight and consistent communication between themselves to avoid 

adverse interactions and ensure the safety of everyone. UAS operations that are autonomous and/or beyond line of 

sight must integrate into this communication flow; however, there is currently no automation infrastructure in place 

to facilitate UAS operations and communication in uncontrolled airspace. To fill this void, NASA has instituted the 

NASA Unmanned Aircraft System (UAS) Traffic Management (UTM) Project.

3 

“The UTM system will enable safe and efficient low-altitude airspace operations by providing services such as 

airspace design, corridors, dynamic geo-fencing, severe weather and wind avoidance, congestion management, 

terrain avoidance, route planning, re-routing, separation management, sequencing, spacing, and contingency 

management. UTM is essential to enable the accelerated development and use of civilian sUAS applications. 

In its most mature form, the UTM system will be developed using autonomicity characteristics, which will 

include self-configuration, self-optimization, and self-protection.”1 

An initial step in realizing this system is to better understand the sUAS to be operating within by formally defining 

mission task elements (MTEs) and their associated requirements for nominal and off-nominal flight. It is through these 

MTEs that the performance of the sUAS will be formally assessed and thereby performance in off-nominal conditions 

may be compared. From there, handling qualities4 and risk may be more thoroughly assessed, and safety assured. 

To this end, a study was conducted to support the ongoing effort to develop, test, and validate MTEs for sUAS, 

specifically multirotor aircraft, in off-nominal conditions. The off-nominal conditions comprised a controlled wind 

environment generated in the NASA LaRC 14x22-ft subsonic wind tunnel. Two multirotor aircraft were free flown in 

the wind stream in March 2019—a feat that has never been accomplished with a multirotor aircraft, and the last fixedwing free flight in this wind tunnel occurred in the 1990s—and exercised through a common set of multirotor MTEs 

that were first introduced in Ref. 5. This builds upon a similar study conducted in nominal conditions in the NASA 

LaRC Autonomy Incubator.6 It is intended that the evaluations described herein will serve to vet the MTEs and the 

process by which they are developed and tested as well as to document off-nominal system performance of the selected 

vehicles. 

This paper discussing this flight testing in controlled wind conditions is organized as follows. The low speed MTEs 

developed for multirotor aircraft performance assessment are described in Section III. The setup for testing in 

controlled wind conditions is outlined in Section IV. The flight test protocol is summarized in Section V. Results and 

discussion are presented in Section VI followed by the conclusions in Section VII. 

III. Mission Task Elements 

The current philosophy of STI and NASA LaRC for the testing and certification process of multirotor sUAS is to 

define the tasks as mission-oriented Mission Task Elements (MTEs).5 This leverages categorizing segments of the 

mission into specific tasks. It is intended that the MTEs be specified in detail, including performance 

requirements. Flight phase categories are defined in terms of the level of precision and aggressiveness required of the 

UAS. Four categories of MTEs using these terms for flight phase include: 

1. Non-Precision, Non-Aggressive – Non-precision tasks that require only a moderate amount of remote 

pilot or autonomous system control fall in this category. 

2. Non-Precision, Aggressive – This category includes the large amplitude maneuvering MTEs that 

emphasize control power over crisp dynamics. Moderate- and large-amplitude maneuvering requirements 

are of primary interest for these MTEs. This category invokes some of the existing control power criteria 

as well as other agility criteria. 

3. Precision, Non-Aggressive – This category includes tasks where considerable precision is required, but 

without the aggressive control activity. The dynamic response requirements for these tasks are less 

stringent than for Precision, Aggressive, but significantly greater than for Non-Precision, NonAggressive. 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

4. Precision, Aggressive – This category includes precision tasks, where an extremely crisp and predictable 

response to control inputs is required. The results of not achieving the required precision are usually 

significant in terms of accomplishing the mission or safety of flight. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

A. Types of Tasks 

For the purposes of the study conducted for this program, a set of four Precision, Non-Aggressive MTEs were 

developed as a basis upon which to build and refine: 1) Precision Hover, 2) Lateral Sidestep, 3) Vertical Reposition, 

and 4) Landing. The objectives, task setup, and requirements are loosely inspired by MTEs for fixed-wing aircraft and 

rotorcraft7 as well as experience of the STI and LaRC team. Testing and data analysis for this study discussed herein 

as well as future testing and analysis will likely indicate that refinements need to be made to more appropriately guide 

the nominal system performance of multirotor sUAS in wind. 

B. Precision, Non-Aggressive MTEs 

1. Precision Hover MTE 

Objectives 

• Evaluate ability to transition from translating flight to a stabilized hover over a target hover 

zone with precision and a reasonable amount of aggressiveness. 

• Evaluate ability to maintain precise position, heading, and altitude over the target. 

Description 

The MTE will start 25 ft aft of the hover board and approximately 15 ft left or right of the hover board center. 

From this starting point, the vehicle will maneuver at a constant altitude of approximately 5 ft and a speed of 

approximately 5 kts to the precision hover point that is 25 ft aft of the hover board center. Attain a stabilized hover in 

the defined target hover zone marked by the hover board. Maintain the stabilized hover at the 25 ft aft location in front 

of the hover board for the time duration specified in the performance requirements. Repeat as needed. 

The center of the hover board will be placed 5 ft above and 25 ft in front of the hover point marker at ground level. 

The hover board will have distinct boundaries indicating desired and adequate performance requirements. 

Desired Performance 

• Attain stabilized hover from the lateral sidestep before exiting the desired region of the hover 

board. 

• ≥30 sec of stabilized hover in desired region. 

• ±1 ft altitude deviation from hover board center. 

• ±1 ft lateral deviation from hover board center. 

• ±1 ft longitudinal (fore/aft) deviation from hover point. 

• ±5° heading deviation from target or hover board. 

• No undesirable motions (bobble, overshoots/undershoots) that impact task performance during 

the transition to hover or stabilized hover. 

Adequate Performance 

• Attain stabilized hover from the lateral sidestep before exiting the adequate region of the target 

and hover board. 

• ≥30 sec of stabilized hover in adequate region. 

• ±2 ft altitude deviation from hover board center. 

• ±2 ft lateral deviation from hover board center. 

• ±2 ft longitudinal (fore/aft) deviation from hover point. 

• ±10° heading deviation from target or hover board. 

• No oscillations that impact system stability of safety of flight during the transition to hover or 

stabilized hover. 

3 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 1: Precision Hover 

2. Lateral Sidestep MTE 

Objectives 

• Assess roll axis and heave axis response during moderately aggressive maneuvering. 

• Identify undesirable coupling between the roll controller and the other axes. 

Description 

From a stabilized hover at an altitude of 5 ft with the longitudinal axis of the multi-rotor sUAS oriented 90 degrees 

to a reference line marked on the ground, initiate a lateral acceleration to approximately 5 kts groundspeed followed 

by a deceleration to laterally reposition the vehicle to a stabilized hover 12.5 ft left of the starting point as indicated 

by another ground marker all while maintaining the initial heading throughout the maneuver. The acceleration and 

deceleration phases shall be accomplished as single smooth maneuvers. The reposition capture is complete when a 

stabilized hover is achieved as indicated by the vehicle position in front of the hover boards, left or right depending 

on course position. 

The center of the (left and right) hover boards will be placed 5 ft above ground and approximately 25 ft apart 

laterally, equally spaced left and right of the starting point. (In the NASA LaRC Autonomy Incubator, the starting 

point was 20 ft aft of the precision hover board.) The hover board will have distinct boundaries indicating desired and 

adequate performance requirements. 

Desired Performance 

• ±1 vertical deviation from hover board center at each capture point. 

• ±1 ft lateral deviation as indicated by the hover board center at each capture point. 

• ±1 ft longitudinal (fore/aft) deviation from ground marker. 

• ±5° heading deviation from reference heading. 

Adequate Performance 

• ±2 vertical deviation from hover board center at each capture point. 

• ±2 ft lateral deviation as indicated by the hover board center at each capture point. 

• ±2 ft longitudinal (fore/aft) deviation from ground marker. 

• ±10° heading deviation from reference heading. 

Figure 2: Lateral Reposition (mini course) 

4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3. Vertical Reposition MTE 

Objectives 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

• Evaluate heave damping, i.e., the ability to precisely control and stop a vertical rate. 

• Evaluate vertical control power. 

• Identify undesirable coupling between collective and the pitch, roll, and yaw axes. 

Description 

From a stabilized hover at an altitude of 5 ft, initiate a vertical ascent of at least 10 ft and stabilize for 5 seconds at 

an altitude of at least 15 ft using an available landmark to stabilize vertical position. Descend back to the initial hover 

position in front of the hover board (5 ft altitude and 25 ft aft of the hover board center) and stabilize for at least 5 

seconds. Maintain initial heading and longitudinal/lateral position throughout the maneuver. Repeat as needed. 

The center of the hover board will be placed 5 ft above and 25 ft in front of the hover point marker at ground level. 

The hover board will have distinct boundaries indicating desired and adequate performance requirements. 

Desired Performance 

• ±1 vertical deviation from start/finish altitude as indicated by the hover board center. 

• ±1 ft lateral deviation as indicated by the hover board center. 

• ±1 ft longitudinal (fore/aft) deviation from ground marker. 

• ±5° heading deviation from ground marker. 

Adequate Performance 

• ±2 vertical deviation from start/finish altitude as indicated by the hover board center. 

• ±2 ft lateral deviation as indicated by the hover board center. 

• ±2 ft longitudinal (fore/aft) deviation from ground marker. 

• ±10° heading deviation from ground marker. 

Figure 3: Bob-up/Down (Vertical Altitude Change) 

4. Landing MTE 

Objectives 

• Evaluate precision control of multi-rotor position during final descent to a precision landing 

point. 

Description 

From an initial altitude of greater than 10 ft as indicated by a hover board or other appropriate landmark, maintain 

an essentially steady descent to a prescribed landing point as marked on the ground by a landing marker while 

maintaining a reference heading. It is acceptable to arrest the sink rate momentarily to make last-minute corrections 

before touchdown. 

5 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Desired Performance 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

• ≤2 fps vertical speed at touchdown.

• ±1 ft longitudinal (fore/aft) deviation from landing marker. 

• ±1 ft lateral deviation from landing marker. 

• ±5° heading deviation from landing marker. 

• Smooth, continuous descent with no undesirable motions that may impact task performance. 

Adequate Performance 

• ≤4 fps vertical speed at touchdown. 

• ±2 ft longitudinal (fore/aft) deviation from landing marker. 

• ±2 ft lateral deviation from landing marker. 

• ±10° heading deviation from landing marker. 

• No system oscillations during landing maneuver. 

Figure 4: Landing 

IV. Testing in Controlled Wind Conditions 

A. NASA Langley 14x22-ft Subsonic Wind Tunnel 

The multirotor MTE development and refinement in a controlled wind environment was conducted indoors in the 

NASA Langley 14x22-ft Subsonic Wind Tunnel. The test section is 14x22 ft with a control room just off the test 

section with a view through a plexiglass panel. To ensure the safety of the personnel positioned within the test section 

as well as the vehicle and tunnel, there were plexiglass panels behind which the personnel stood during testing, and 

the sUAS was affixed to the floor of the test section via a retracting 3/16-in cable. 

Figure 5: 14x22-ft Subsonic Wind Tunnel 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

B. Test Aircraft 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

Two aircraft were used for this test for both the checkout and the evaluation test phases: a Tarot X6 hexacopter 

(Figure 6a) and a Tarot 650 Sport quadcopter (Figure 6b). These aircraft are owned and maintained by NASA LaRC 

and flown for many programs on campus. 

The Tarot X6 is a 960 class hexacopter with 11-in. props, 5008 340kV motor, and 15-min flight time. The Tarot 

650 Sport is a 650 class quadcopter with 10-in. props, 4114-320kV motor, and 12-min flight time. Both aircraft are 

outfitted with standard Pixhawk autopilots. Vicon tracking nodes were affixed to the aircraft for precision, indoor 

position and attitude tracking. For safety purposes, a 13-ft tether was attached to each aircraft with small weights on 

the end. This tether was fed through a hole in the wind tunnel floor and acted as a retractable safety line. 

The battery packs are mounted to the bottom of the central node of the aircraft. For all of the MTEs, the pilots used 

these packs as their reference point when flying, meaning the pilot attempted to fly the battery pack within the desired 

regions as indicated by the hover boards described in the next section. 

a) Tarot X6 b) Tarot 650 Sport 

Figure 6: Multirotor Test Vehicles 

C. Course Layout 

The course was defined by a set of hover boards that served as visual cues for the desired and adequate performance 

bounds for the tasks (see Figure 7). The inertial (0,0,0) point of the space was defined as the hole in wind tunnel floor 

through which the safety tether fed. This point was 11.5 ft back from the leading edge of the wind tunnel test section 

and 18 in. right of center when facing the wind tunnel mouth. A ±1-ft box was marked by tape around this point. 

The center of the hover boards marked by a “+” indicated the (0,0) point of the performance bound. The inner box

indicates a ±1-ft bound, and the outer box indicates a ±2-ft bound. Three boards were affixed to the wall directly across 

from the pilot and in line with the (0,0,0) point to mark the hover course (bottom board only), the right capture of the 

lateral sidestep course (bottom board only), the vertical reposition course (top and bottom boards), and the landing 

course (top board only). The hover board for the left capture of the lateral sidestep course was mounted to a crane. 

This crane was positioned such that a large enough sidestep was performed but did not force the pilot to fly the aircraft 

at the full length of the safety tether. 

Figure 7: MTE Course Layout 

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D. Data Collection 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

Three sources of data were collected for these tests: Vicon, Pixhawk, and video. A Vicon system was installed in 

the open test section of the wind tunnel along the perimeter of the test area shown in Figure 7. The Vicon system was 

a Vantage V5, which utilizes five cameras with 5-megapixel resolution at 2000 Hz to achieve high precision position 

and attitude tracking. This data was used for performance assessment of the wind tunnel tests. 

The Pixhawk autopilots installed on the aircraft record GPS, IMU, pilot command, and control deflection data. 

The GPS data were not accurate for these tests in the indoor wind tunnel and cannot be used to measure performance. 

The IMU, pilot command, and control deflection data will be valuable in understanding the pilot-vehicle system and 

how that affects performance, but that discussion is outside the scope of this paper. 

Two videos were recorded for all flights. One camera was set up just behind the shoulder of the pilot behind the 

blast screen with a view of the whole course. This angle captured pilot perception and line-of-sight. The second camera 

was set up ~40 ft downstream of the pilot with a view of the whole course. These videos served to verify run log 

information, pilot visual cues, and pilot comments recorded during the tests. 

V. Flight Test Summary 

The wind tunnel entry consisted of one day of checkout testing and two days of evaluations. The checkout testing 

consisted of several tasks: 

• Installing the Vicon system and ensuring it was not unduly affected by the wind tunnel when 

in operation. 

• Marking the (0,0,0) point on the floor of the wind tunnel. 

• Positioning hover boards to define the course layout within the constraints of the wind tunnel 

and safety requirements. 

• Testing each aircraft with the safety tether affixed and marking the tether in 3-ft sections for 

additional cueing. 

• Testing each aircraft with the wind tunnel on with speeds up to 15 mph. 

It was during the wind on tests that an interesting phenomenon occurred. The standard method of flying multirotor 

aircraft with Pixhawk autopilots is in Altitude Hold mode as this usually decreases pilot workload. When this wind 

tunnel runs at low speeds, such as 5 mph, there is some swirling of the air along the boundaries of the tunnel. These 

swirls cause local changes in barometric pressure. The Altitude Hold mode uses barometric pressure as its feedback 

signal. This caused the aircraft motors to surge rhythmically as the autopilot attempted to maintain altitude, which 

induced a marked increase in pilot workload. It was quickly determined that flying in the wind tunnel with Altitude 

Hold mode was not conducive to this testing, so Stability Hold mode was used for all evaluation testing. 

The evaluation consisted of testing the four MTEs at six windspeeds with two pilots and two aircraft. The full test 

matrix is listed below in Table 1. Flight time was constrained by the batteries on the aircraft. Therefore, the test 

procedure was designed accordingly. For each flight, the wind tunnel was brought up to the speed at which the tasks 

were to be flown. The first pilot took off and performed three hover MTEs, three lateral sidestep MTEs, three vertical 

reposition MTEs, and three landing MTEs. After the last landing, the first pilot disarmed the aircraft and handed the 

controls to the second pilot. The second pilot armed the aircraft, took off, and performed the same sequence of MTEs. 

After the last landing, after ~10 minutes of total flight time, the aircraft was disarmed, and the wind tunnel was shut 

down. The spent battery was replaced with a fresh one, and the process was repeated at the next test point. The same 

procedure was followed for both aircraft. 

8 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1: Test Matrix 

Aircraft Pilot Windspeed MTEs 

(mph) 

Tarot X6 1 0 H, LS, VR, L* 

Tarot X6 2 0 H, LS, VR, L 

Tarot X6 1 5 H, LS, VR, L 

Tarot X6 2 5 H, LS, VR, L 

Tarot X6 1 10 H, LS, VR, L 

Tarot X6 2 10 H, LS, VR, L 

Tarot X6 1 15 H, LS, VR, L 

Tarot X6 2 15 H, LS, VR, L 

Tarot X6 1 20 H, LS, VR, L 

Tarot X6 2 20 H, LS, VR, L 

Tarot X6 1 25 H, LS, VR, L 

Tarot X6 2 25 H, LS, VR, L 

Tarot 650 Sport 1 0 H, LS, VR, L 

Tarot 650 Sport 2 0 H, LS, VR, L 

Tarot 650 Sport 1 5 H, LS, VR, L 

Tarot 650 Sport 2 5 H, LS, VR, L 

Tarot 650 Sport 1 10 H, LS, VR, L 

Tarot 650 Sport 2 10 H, LS, VR, L 

Tarot 650 Sport 1 15 H, LS, VR, L 

Tarot 650 Sport 2 15 H, LS, VR, L 

Tarot 650 Sport 1 20 H, LS, VR, L 

Tarot 650 Sport 2 20 H, LS, VR, L 

Tarot 650 Sport 1 25 H, LS, VR, L 

Tarot 650 Sport 2 25 H, LS, VR, L 

*

H = Hover, LS = Lateral Sidestep, VR = Vertical Reposition, L = Landing 

VI. Results and Discussion 

For brevity, results that exemplify performance of each MTE and summary results are presented. Representative 

tasks at 5 mph and 25 mph are highlighted because they best show the effect of the controlled wind environment. 

Performance of the tasks did not change significantly from 0 to 5 mph while there was noticeable degradation in 

performance as windspeed increased to 25 mph. Runs for Pilot 1 flying the Tarot X6 are shown for the hover, lateral 

sidestep, and vertical reposition examples as he tended toward calm purposefulness in his piloting style with this 

aircraft regardless of windspeed and thus yielded clean results. The landing example is for Pilot 2 as Pilot 1 chose not 

to attempt landings at 25 mph. These examples set the basis for the discussion in the subsequent subsection. From this 

collection of data, it is possible to assess the performance of each MTE and determine if and how the MTEs should 

be improved. 

The full NASA report, reduced data, and raw data is available in Ref. 8. From this collection of data, it is possible 

to assess the performance of each MTE and determine if and how the MTEs should be improved. 

A. Performance Metrics 

Based on the MTE definitions in Section III.B, performance metrics can be defined based on the vehicle’s position

(i.e., X, Y, and Z coordinates), attitude (i.e., roll, pitch and heading angles), and velocity depending on the MTE. Since 

the Vicon data does not include velocity, it was derived by differentiating aircraft position. Performance metrics 

include % Desired performance and % Adequate performance. 

This metric is calculated by assessing each data point over the test performance range or scoring to determine if it 

falls within the desired or adequate bounds and dividing the number of points within each by the total number of data 

points. If the pilot keeps the aircraft within the desired bounds more than 50% of the time for the task, then the overall 

9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

task performance is considered desired. If less than 50% is within the desired bounds, but more than 50% is within 

the adequate bounds, then task performance is adequate. If less than 50% is within the adequate bounds, then the pilot 

fails in task performance. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

In the plots in the following subsection, the desired performance region is defined by an opaque grey region 

outlined in solid black lines, the adequate performance region is defined by dashed lines, the flight data is shown in 

blue, and the performance test range or scoring region for the run is defined by dashed red lines. 

B. MTE Example Results 

1. Precision Hover 

To execute the precision hover task, pilots were instructed to start the vehicle and begin hovering at 5 feet. The 

pilot then executed a step to the left while maintaining the longitudinal position and altitude. The pilot immediately 

stepped the vehicle back to the original lateral position and hovered above the origin for 5-10 seconds. This MTE was 

repeated three times before proceeding to the next MTE (see example in Figure 1). 

A comparison of X position, Y position, altitude or Z position, and heading for the hover MTE performed at a 

windspeed of 5 mph and a windspeed of 25 mph is shown in Figure 8. The primary task for this MTE was to capture 

and hold a stable Y and Z position in front of the pilot. The secondary task was to maintain X position and heading. 

From this example, it is easy to pick out the three hover tasks completed by this pilot in the Y position data in 

Figure 8c and d, the first of which is indicated by the dashed red lines. Qualitatively, there is slightly more activity in 

these states for the hover MTEs performed at the higher windspeed, but the pilot was generally able meet the 

performance requirements. 

This observation is supported by the performance data listed in Table 2 and Table 3 for the indicated run. The pilot 

had desired performance in both the primary task and secondary task for both windspeeds. However, there is a 

significant decrease in % Desired in X position and heading at the 25-mph windspeed. This is attributed to the lack of 

sufficient cueing for longitudinal position and heading on the test course. It is challenging to maintain these states 

with the available cues at the relatively lower workload of the lower windspeed. The higher windspeed increased pilot 

workload according to pilot comments and the aircraft attitudes not shown here (namely roll attitude), which resulted 

in less attention paid to the secondary task. 

10 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

a) X Position, 5 mph b) X Position, 25 mph 

c) Y Position, 5 mph d) Y Position, 25 mph 

e) Altitude, 5 mph f) Altitude, 25 mph 

g) Heading, 5 mph h) Heading, 25 mph 

Figure 8: Hover MTE Example 

11 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 2: Hover Performance, 5 mph 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

Pilot: 1

Windspeed: 5 mph

Run: 2 Desired (%) Adequate (%)

X Position 91.63 100.00

Y Position 99.94 100.00

Z Position 94.44 100.00

Heading Angle 100.00 100.00

Task: Hover

Vehicle: Tarot X6

Table 3: Hover Performance, 25 mph 

Pilot: 1

Windspeed: 25 mph

Desired (%) Adequate (%)

X Position 75.45 100.00

Y Position 99.94 100.00

Z Position 100.00 100.00

Heading Angle 80.13 100.00

Task: Hover

Vehicle: Tarot X6

2. Lateral Sidestep 

For the lateral sidestep MTE, the pilots were instructed to fly the vehicle laterally in one direction, capture and 

hold a static position, then sidestep in the other direction to capture and hold the original position. The pilot first flew 

the aircraft to the hover board directly in front of him and in line with the takeoff point and then sidestepped towards 

a hover board to the left of the takeoff point resulting in a roughly 10-ft sidestep. The lateral sidestep was then repeated 

three times (see example in Figure 2). Each capture lasted 5-10 seconds with the test director counting to five as aural 

cue of time in position. 

A comparison of X position, Y position, altitude or Z position, and heading for the lateral sidestep MTE performed 

at a windspeed of 5 mph and a windspeed of 25 mph is shown in Figure 9. Similar to the hover MTE, the primary task 

for this MTE was to capture and hold a stable Y and Z position, and the secondary task was to maintain X position 

and heading. 

From this example, it is again easy to pick out the three sidestep tasks completed by this pilot in the Y position 

data shown in Figure 9c and d, of which the third left sidestep is indicated by the dashed red lines. Qualitatively, the 

right step, for which the capture hover board is directly in front of the pilot, is performed more cleanly than the left 

according to these two subfigures. There is an offset in X position and more deviation in altitude for the 5-mph case. 

This could be due to the pilot focusing largely on maintaining lateral position especially at the left hover board and 

not correcting for the offset. 

For the 25-mph case, the pilot doesn’t appear to be able to capture hover board when sidestepping left well. A

probable cause of this issue is the pilot’s perspective of that hover board (refer to Figure 1). An offset in X position is 

difficult to perceive at that angle, thus making it possible that the pilot believes he has captured the hover in the desired 

region based on his line of sight. When Y position is corrected for this X offset, as shown in Figure 10, his performance 

appears to be well within the adequate performance bounds if not the desired performance bounds. 

These observations are reflected in the performance data listed in Table 4 and Table 5 for the indicated run. 

Heading and Z position have desired performance for both windspeeds. The pilot failed X position performance for 

the MTE flown at 5 mph with a 23.9% Adequate. The Y position performance increased significantly for both 

windspeeds with a change from an adequate performance of 69.53% Adequate to a desired performance for the MTE 

performed at 25 mph. 

12 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

a) X Position, 5 mph b) X Position, 25 mph 

c) Y Position, 5 mph d) Y Position, 25 mph 

e) Altitude, 5 mph f) Altitude, 25 mph 

g) Heading, 5 mph h) Heading, 25 mph 

Figure 9: Lateral Sidestep MTE Example 

13 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

a) 5 mph b) 25 mph 

Figure 10: Lateral Sidestep Corrected Y Position 

Table 4: Lateral Sidestep Performance, 5 mph 

Pilot: 1

Windspeed: 5 mph

Run: 1 Desired (%) Adequate (%)

X Position 2.78 23.90

Y Position 74.82 100.00

Y Position (Corrected) 93.40 100.00

Z Position 74.59 100.00

Heading Angle 100.00 100.00

Task: Side Step

Vehicle: Tarot X6

Table 5: Lateral Sidestep Performance, 25 mph 

Pilot: 1

Windspeed: 25 mph

Desired (%) Adequate (%)

X Position 71.25 100.00

Y Position 24.60 69.53

Y Position (Corrected) 100.00 100.00

Z Position 100.00 100.00

Heading Angle 100.00 100.00

Task: Side Step

Vehicle: Tarot X6

3. Vertical Reposition 

For the vertical reposition MTE, the pilots were instructed to takeoff and stabilize in a hover for 5 seconds relative 

to the hover board directly in front of him. The pilots were then instructed to fly the vehicle up to a hover board placed 

~10 feet above the lower hover board. Due to the perspective of the pilot, this placed the aircraft ~10 ft above the 

ground. As such the requirement for the MTE was modified to 10 ft above the ground from the original 15 ft. This 

also kept the upper capture point well within the wind tunnel test section, which is 14 ft in height. Once at this altitude 

the pilot stabilized the vehicle in a hover for 5 seconds as counted by the test director and then descended back to the 

original hover (see example in Figure 3). 

A comparison of X position, Y position, altitude or Z position, and heading for the vertical reposition MTE 

performed at a windspeed of 5 mph and a windspeed of 25 mph is shown in Figure 11. Again, the primary task for 

this MTE was to capture and hold a stable Y and Z position, and the secondary task was to maintain X position and 

heading. 

From this example, the three vertical reposition MTEs completed by this pilot can be deduced from the Z position 

data shown in Figure 11e and f, of which the third ascending vertical reposition is indicated. Maintaining X position 

and heading appears to be a challenging secondary task for this MTE. Y position is generally maintained well, and the 

descending vertical reposition is a more cleanly performed task in altitude performance as is expected from previous 

observations. 

14 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

a) X Position, 5 mph b) X Position, 25 mph 

c) Y Position, 5 mph d) Y Position, 25 mph 

e) Altitude, 5 mph f) Altitude, 25 mph 

g) Heading, 5 mph h) Heading, 25 mph 

Figure 11: Vertical Reposition MTE Example 

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

However, like the left lateral sidestep, the ascending vertical reposition in which the upper hover board is used as 

the visual cue for position capture and hold appears more challenging. The altitude of the vehicle was highly influenced 

by the longitudinal position of the vehicle and the pilot’s line of sight (refer to Figure 3) much like the lateral position 

was influenced for the lateral sidestep MTE. An offset in X position is difficult to perceive at that angle, thus making 

it possible that the pilot believes he has captured the hover in the desired region based on his line of sight. When Z 

position is corrected for this X offset, as shown in Figure 12, the pilot’s performance appears to be well within the

adequate performance bounds if not the desired performance bounds for both cases, though this does not appear to 

have had as significant an effect as the did the Y position correction for the lateral sidestep MTE performance. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

These observations are reflected in the performance data listed in Table 6 and Table 7 for the indicated run. 

Heading and Y position have desired performance for both windspeeds. The pilot barely achieved desired performance 

in X position at 5 mph with a 53.4% Desired, and he only achieved adequate performance in X position at 25 mph 

with a 71.23% Adequate. For both windspeeds, the corrected Z position resulted in a slightly worse performance 

metric, decreasing from 93.2% to 88.78% Desired for the 5-mph case and from 96.44% to 92.68% Desired for the 25-

mph case. This suggests that the pilot is better able to regulate X position for this MTE and a correction in Z position 

performance is not required. 

a) 5 mph b) 25 mph 

Figure 12: Vertical Reposition Corrected Z Position 

Table 6: Vertical Reposition Performance, 5 mph 

Pilot: 1

Windspeed: 5 mph

Desired (%) Adequate (%)

X Position 53.40 100.00

Y Position 100.00 100.00

Z Position 93.20 100.00

Z Position (Corrected) 88.78 100.00

Heading Angle 60.29 100.00

Task: Vertical Reposition

Vehicle: Tarot X6

Table 7: Vertical Reposition Performance, 25 mph 

Pilot: 1

Windspeed: 25 mph

Desired (%) Adequate (%)

X Position 38.95 71.23

Y Position 100.00 100.00

Z Position 96.44 100.00

Z Position (Corrected) 92.68 100.00

Heading Angle 100.00 100.00

Task: Vertical Reposition

Vehicle: Tarot X6

4. Landing 

To execute the landing task, pilots were instructed to fly the vehicle above 10 ft, then gently land the vehicle at the 

same spot from which they took off, minimizing lateral and longitudinal deviations (refer to Figure 4). They were also 

required to limit vertical velocity at touchdown to ≤ 2 ft/s. Each pilot executed this MTE three times per windspeed 

per vehicle (see example in Figure 4). There was no requirement to capture or maintain a certain altitude as the focus 

was on the landing position. 

16 
Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

a) X Position, 5 mph b) X Position, 25 mph 

c) Y Position, 5 mph d) Y Position, 25 mph 

e) Altitude, 5 mph f) Altitude, 25 mph 

g) Heading, 5 mph h) Heading, 25 mph 

Figure 13: Landing MTE Example 

17 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

A comparison of X position, Y position, altitude or Z position, and heading for the landing MTE performed at a 

windspeed of 5 mph and a windspeed of 25 mph is shown in Figure 13. Vertical velocity is shown in Figure 14. The 

primary task for this MTE is maintaining X and Y position while modulating vertical velocity. The secondary task 

was to maintain heading. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

From this example, each landing MTE can be determined from the altitude of the aircraft. The beginning of the 

task is roughly an apex in Z position, and the ending is when altitude is steady at roughly the height of the battery 

pack of the aircraft. Qualitatively, there are stark differences between the MTE performance for the two windspeeds. 

In general, the states of the aircraft in 5 mph are smoother than in 25 mph winds. This is especially stark in the vertical 

velocity, though the pilot does keep the vertical velocity below the 2 ft/s requirement. X position is difficult to maintain 

in the descent at both windspeeds, but maintaining Y position and heading significantly degraded at windspeed of 25 

mph. 

These observations are supported by the performance data listed in Table 8 and Table 9. X position performance is 

only 26.0% and 54.5% Desired for the 5-mph and 25-mph cases, respectively. For the 5-mph case, Y position, vertical 

velocity, and heading were all perfectly maintained within the desired performance bounds. For the 25-mph case, Y 

position and heading were only maintained within the adequate performance bounds and had 44.31% and 12.48% 

Desired performance, respectively. 

a) 5 mph b) 25 mph 

Figure 14: Landing Vertical Velocity 

Table 8: Landing Performance, 5 mph 

Pilot: 2

Windspeed: 5 mph

Desired (%) Adequate (%)

X Position 26.00 100.00

Y Position 100.00 100.00

Landing Speed 100.00 100.00

Heading Angle 100.00 100.00

Task: Landing

Vehicle: Tarot X6

Table 9: Landing Performance, 25 mph 

Windspeed: 25 mph

Desired (%) Adequate (%)

X Position 54.50 89.43

Y Position 44.31 100.00

Landing Speed 100.00 100.00

Heading Angle 12.48 96.43

Vehicle: Tarot X6

C. Summary Results and Discussion 

The average % Desired performance of each MTE performed at each windspeed by both pilots on both aircraft are 

illustrated in the subfigures of Figure 15. 

For the Hover MTE in Figure 15a and b, the pilots had consistently high performance scores for altitude and lateral 

position (red and blue symbols in Figure 15, respectively). For these two performance measures, the pilots maintained 

an adequate performance for all maneuvers performed and almost always met the desired performance requirements 

18 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

as well. The most difficult task was maintaining the aircraft’s longitudinal position in the desired region. They also 

struggled to keep the heading angle within the desired bounds, especially at higher windspeeds. 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

In general, the pilots flew better with the Tarot X6 aircraft. There was more dispersion in the % Desired for the 

pilots’ performances with the Tarot 650 Sport with more of the metrics falling below the 50% threshold to be deemed 

an Overall Desired performance. In addition, there were two metrics for the pilots’ performances with the Tarot 650 

Sport that fell below the 50% threshold in % Adequate as well. 

For the Lateral Sidestep MTE, both pilots excelled at maintaining a desired lateral position for each Sidestep 

maneuver as indicated by the consistently high % Desired in Figure 15c and d. Although both pilots rarely maintained 

smooth altitude tracking, they were consistent at keeping the aircraft within the adequate bounds throughout the 

maneuvers. They consistently maintained a % Desired performance above 80% for both aircraft save for a couple test 

points, as seen in Figure 15. However, the pilots struggled to maintain longitudinal positioning within the desired 

bounds (black symbols in Figure 15) and only achieved an Overall Desired performance for ~50% of the test points. 

The pilots had moderate success maintaining the aircraft’s heading angle within the desired bounds and achieved an

Overall Desired performance for 65% of the test points. 

Comparing the two vehicles, the only significant difference in performance is the heading angle. Considering the 

heading performance data shown in Figure 15c and d, the average % Desired for the Tarot X6 are consistently higher 

than those for the Tarot 650 Sport. With the Tarot 650 Sport, the pilots only achieved an Overall Desired for heading 

angle in 47% of the test points. With the Tarot X6, the pilots achieved an Overall Desired performance for heading 

angle in 87% of the test points. Pilot 2 appeared to struggle even more with the Tarot 650 Sport with several average 

% Desired for heading angle below 10%. 

For the Vertical Reposition MTE in Figure 15e and f, both pilots excelled at maintaining a desired lateral position 

(blue symbols in Figure 15) for the duration the vertical reposition maneuvers for both aircraft. For all but one test 

point the average % Desired for Y Position is 100.00%. The pilots also consistently maintained altitude within the 

desired performance bounds, achieving an average Overall Desired performance for the Z Position for all of the test 

points. This suggests that the effects of longitudinal position on altitude for the Vertical Reposition MTE are not as 

significant as for Y Position in the Lateral Sidestep MTE. 

As with the Lateral Sidestep and Hover MTEs, the pilots struggled to maintain a steady longitudinal position within 

the desired performance bounds. The pilots only had an average Overall Desired performance in X Position for 55% 

of the test points. Pilot 2 again struggled with heading angle and had an average Overall Desired performance in 

heading angle for only 50% of the test points, while Pilot 1 had an Overall Desired performance for all of test points. 

Like the Lateral Sidestep MTE, most of Pilot 2’s heading angle struggles were while flying the Tarot 650 Sport. 

Finally, for the Landing MTE in Figure 15g and h, controlling the vertical velocity when landing was fairly easy 

for the pilots with both vehicles. The performance assessment of landing vertical velocity is a discrete metric in which 

the pilot had Desired, Adequate, or Failed performance. For these data, the failure condition does not appear because 

the pilots achieved desired performance for all maneuvers and never exceeded the 4-ft/s failure limit. 

The pilots once again performed poorly with regard to longitudinal position during this MTE. The pilots achieved 

an average Overall Desired performance in longitudinal position for 72% of the test points. Pilot 1 was less capable 

that Pilot 2 in maintaining longitudinal position and only achieved an average Overall Desired performance in 

longitudinal position for 67% of the test points. The pilots had greater success in maintaining their heading angle 

during this MTE and were able to maintain an average Overall Desired heading angle performance for all but four of 

the test points. There was no apparent difference in performance measures between the two aircraft. 

19 
-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

Legend X Position Y Position

Z Position Heading

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

 

 

 

 

 

a) Hover, Tarot X6 b) Hover, Tarot 650 Sport 

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

Legend X Position Y Position

Z Position Heading

c) Lateral Sidestep, Tarot X6 d) Lateral Sidestep, Tarot 650 Sport 

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

Legend X Position Y Position

Z Position Heading

e) Vertical Reposition, Tarot X6 f) Vertical Reposition, Tarot 650 Sport 

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

-5 0 5 10 15 20 25 30 Windspeed (mph)

0

10

20

30

40

50

60

70

80

90

100

% Desired

Legend X

Y

Landing Speed Heading

g) Landing, Tarot X6 h) Landing, Tarot 650 Sport 

Figure 15: Average % Desired Performance ( - Pilot 1,  - Pilot 2) 

20 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

VII. Conclusions 

Downloaded by NASA AMES RESEARCH CENTER on January 13, 2020 | http://arc.aiaa.org | DOI: 10.2514/6.2020-1507 

The purpose of this flight test in controlled wind conditions was to determine the effectiveness of the missionoriented approach as a means to define sUAS off-nominal performance. NASA LaRC’s 14x22-ft subsonic wind tunnel 

was employed to generate the controlled wind conditions in which small multirotor UAS were free flown. The MTEs 

evaluated in the test program were precision hover, lateral sidestep, vertical reposition, and landing. Two multirotor 

aircraft were flown by two pilots through these MTEs at six windspeeds ranging from 0 mph to 25 mph. Performance 

metrics constituted % Adequate and % Desired measures based on requirements defined in the MTE descriptions. 

From the qualitative results that included pilot debrief questionnaires and the quantitative task performance results 

from the Vicon data, it was clearly demonstrated that the MTEs met the defined objectives. Aggregate results show 

that pilot performance does not vary inversely with increasing windspeed, which indicatesthe validity for these MTE’s

in controlled wind environment. Pilot feedback rated the MTEs as effective for simulating operational tasks. 

From this study, several conclusions can be drawn. The general procedure utilized for this test to evaluate small 

multirotor aircraft through a set of precision non-aggressive MTEs within the constraints of the wind tunnel test section 

and safety requirements is sound. The pilots were generally able to perform the MTEs with at least adequate 

performance, though performance degraded at the higher windspeeds. This suggests that upper wind limits could be 

defined for this size aircraft. The pilots often had difficulty maintaining X position within the performance bounds 

due to inadequate longitudinal visual cueing especially at higher windspeeds. Pilot feedback suggested that better 

longitudinal cueing is required. However, an alternative conclusion is that longitudinal cueing in a real world 

environment will likely be even less clear, therefore certifying agencies must either not consider that important or not 

allow piloted sUAS in environments in which precise longitudinal positioning is required. 

Acknowledgements 

The authors acknowledge the support of the NASA LaRC point of contact Steve Riddick without whom this wind 

tunnel entry would not have happened. Furthermore, the authors acknowledge the pilots, the Vicon support team, and 

the 14x22-ft Subsonic Wind Tunnel team led by Frank Quinto who made the free flight testing and data gathering a 

seamless process. The findings and conclusions in this paper are those of the authors and do not necessarily represent 

the views of the funding agency. 

References 

[1] Belcastro, C. M., Newman, R. L., Evans, J. K., Klyde, D. H., Barr, L. C., and Ancel, E., “Hazards Identification and Analysis

for Unmanned Aircraft System Operations,” presented at the AVIATION Forum 2017: 17th AIAA Aviation Technology, 

Integration, and Operations Conference, Denver, CO, 5-9 June 2017. 

[2] Robinsion III, J. E., Johnson, M., Jung, J., Kopardekar, P., Prevot, T., and Rios, J., Unmanned Aerial System Traffic 

Management Concept of Operations, NASA TM-2014-000000, December 2015. 

[3] Kopardekar, P., Rios, J., Prevot, T., Johnson, M., Jung, J., and Robinsion III, J. E., “Unmanned Aircraft System Traffic 

Management (UTM) Concept of Operations,” AIAA 2016-3292 presented at the 16th AIAA Technology, Integration, and 

Operations Conference, Washington, DC, 13-17 June 2016. 

[4] Klyde, D. H., Schulze, P. C., Mitchell, D. G., and Alexandrov, N., “Development of a Process to Define Unmanned Aircraft

Systems Handling Qualities,” AIAA 2018-0299 presented at the AIAA SciTech Forum 2018: AIAA Atmospheric Flight 

Mechanics Conference, Kissimmee, FL, 8-12 Jan. 2018. 

doi: 10.2514/6.2018-0299 

[5] Belcastro, C. M., Klyde, D. H., Logan, M. J., Newman, R. L., and Foster, J. V., “Experimental Flight Testing for Assessing 

the Safety of Unmanned Aircraft System Safety-Critical Operations,” AIAA 2017-3274 presented at the AVIATION Forum 

2017: 17th AIAA Aviation Technology, Integration, and Operations Conference, Denver, CO, 5-9 June 2017. 

doi: 10.2514/6.2017-3274 

[6] Gonzalex, E., Klyde, J. D., Klyde, D. H., and Lampton, A. K., “Multirotor Flight Test Results for Safety Critical Testing using 

Common Mission Task Element Sets,” STI-WP-1460-2, 1 Aug. 2018. 

[7] Anon., “Aeronautical Design Standard, ADS-33E-PRF, Performance Specification, Handling Qualities Requirements for 

Military Rotorcraft,” U.S. Army Aviation and Missile Command, March 21, 2000.

[8] Lampton, A. K., Klyde, D. H., Swaney, T., and Prince, T., Safety Critical Testing using Common Mission Task Element Sets: 

Multirotor Flight Testing in Controlled Wind Conditions, STI-TR-1460-1, 30 June 2019. 

21