Website

RobotTestMethods.nist.gov

Test Director

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Sponsor:

Systems Engineering & Standards Division

Science and Technology Directorate 

U.S. Department of Homeland Security

Email

RobotTestMethods@nist.gov

Version 2020A

Test Methods for Evaluating Aerial Drones

Safety | Capabilities | Proficiency

RobotTestMethods.nist.gov

Pocket Guide for Aerial Drones

Tests Lanes

SENSOR

Visual and Thermal Sensor Tests 

Comprehensive and Quick

Comprehensive tests provide statistical significance. 

Small numbers of tests on panels allow for rapid testing of sensing 

capabilities in different directions and environments. 

Thermal targets can be implemented using laser or waterjet cutting, 

or by drilling holes to minimize fabrication cost and complexity. 

24 
Beams every 1.2 m (4 ft)

~90 m (300 ft)

Sensor Test Lanes

Evaluate capabilities and proficiency

2 23

The Sensor Test Lanes measure the ability of remote pilots and aerial 

drones to inspect objects and observe the environment. They are 

quick, operationally informative field tests, based on more accurate 

but more time consuming tests. 

These tests can be performed outdoors or indoors to control lighting, 

weather, and access to GPS. Various hover altitudes measure 

performance at different distances from objects of interest. 

Performance at the limit of line-of-sight and non-line-of-sight radio 

range is also measured. 

These tests also form 

the basis of artefacts, 

such as the Concentric C 

optotypes, embedded within 

the Open, Obstructed, and 

Confined test lanes and scenarios. 

Stability and Endurance

Hover Stability: Measure the stability of the drone at Hover or 

High Hover altitudes (p6). 

1. Take off and hover at the chosen altitude. 

2. Align with and observe the upward pointing bucket for 30 seconds. 1 

point is awarded if full alignment is maintained for the entire 

duration. 

3. Align with and observe a/the 45 degree angled bucket for 30 seconds. 

1 point is awarded if full alignment is maintained for the entire 

duration. 

4. Repeat (2) and (3) 10 times each and record the total number of 

points awarded, out of a possible maximum of 20. 

Endurance and Return Home: Measure the battery life of the 

drone, and its behavior and accuracy when returning to home. 

1. Record the start time or take a picture of the clock. Set the drone 

home point to the launch pad. 

2. Take off and hover 6 m (20 ft) over the upward facing bucket, 

aligning with it and the angled 6 m (20 ft) bucket. 

3. Fly at speed straight over the 90 m (300 ft) angled

bucket stand. 

4. Turn and fly at speed straight back to the upward facing bucket, 

aligning with it and the angled 6 m (20 ft) bucket. 

5. Climb at speed to 90 m (300 ft) over the upward facing bucket, 

aligning with it and the angled 90 m (200 ft) bucket.

6. Descend to 6 m (20 ft) over the upward facing bucket, 

aligning with it and the angled 6 m (20 ft) bucket. 

7. Repeat from step 2 until the drone automatically 

returns home.

8. Record the number of laps, flight time, and 

if the drone landed with at least one 

ground contact in the 60 cm (2’) 

diameter landing circle. 

This work was developed by the NIST team, which includes Adam Jacoff, Raymond Sheh, 

Kamel Saidi, Alex Fraley, Kenny Kimble, and Ann Virts. Dozens more people have 

contributed to the development and validation of these test methods. They include FEMA 

urban search and rescue task force teams, firefighters, law enforcement, collaborating 

test facilities, other civilian and military organizations, and commercial manufacturers.
3

Acuity and 

Field of View

(Visual, Thermal)

Motion Perception 

and Tracking

Pointing and 

Zooming

Tests for Different Aspects of Sensing

Video and 

Control Latency

Color and Hazmat

Audio Tests

Audio, Stability, 

Endurance and 

Return Home, 

Specialized Tests

Two-Way Audio: For drones equipped with speakers and/or 

microphones. Measure the ability of the drone to maintain twoway voice communication between the operator station and 

people in the environment. 

1. Place a speaker on the ground, 6 m (20 ft) from the landing 

pad. Volume should be set such that the sound level at the 

drone, due to the speaker, is 75-80 dB.

2. Take off from the landing pad and hover at 3 m (10 ft). 

3. Listen at the operator station as alpha-numeric codes are 

played through the speaker. Write the codes on the form 

and total up the number of correctly heard codes. 

4. Place the speaker at the operator station, close to the 

microphone. Set the volume such that the microphone is 

not overloaded. 

5. Listen from 6 m (20 ft) in front of the hovering drone as 

alpha-numeric codes are played through the speaker. Write 

the codes on the form and total up the number of correctly 

heard codes. 

Noise Emission: Using a sound 

pressure level meter, measure the 

sound level caused by operating the 

drone at different altitudes. 

1. Measure baseline sound level with 

drone switched off. 

2. Hover at each designated altitude 

for at least 15 seconds. 

3. Write the average sound level at 

each height. 

(dB)

(dB)

(dB)

(dB)

Baseline Sound Level (dB): ________

22

90 m

(300 ft)

60 m

(200 ft)

30 m

(100 ft)

12 m

(20 ft)
4 Measurements and the Mission

Accurately communicate mission requirements

and system capabilities

Measurements Relate to Requirements:

Measurement Science, through the Standard Test Methods, enables 

the needs of different applications and users to be expressed in a 

common measurement language that unambiguously relates their 

requirements to system capabilities. 

Embedding Measurements: Placing visual acuity targets with 

operationally significant objects of interest allow users to relate 

their requirements to measurable quantities. From an operational 

distance, what level of acuity is needed to tell between hazmat 

labels, read a license plate or

gauge, or tell a gun from a 

cellphone? 

SENSORS | VIDEO AND CONTROL LATENCY

Circle one per line. 

Lit Dark

Wide Zoom

Visible Thermal

Interface Streamed

Camera name: _____________

High Speed Recording Camera Framerate: ____________

Video Latency: 

Frames between flash directly observed and observed through drone: 

1: ______ frames 6: ______ frames

2: ______ frames 7: ______ frames

3: ______ frames 8: ______ frames

4: ______ frames 9: ______ frames

5: ______ frames 10: ______ frames

Average number of frames: ______ frames

Divided by framerate = ______ seconds average video latency. 

Control Latency: 

Frames between command directly observed and the drone moving: 

1: ______ frames 6: ______ frames

2: ______ frames 7: ______ frames

3: ______ frames 8: ______ frames

4: ______ frames 9: ______ frames

5: ______ frames 10: ______ frames

Average number of frames: ______ frames

Divided by framerate = ______ seconds average control latency. 

Statistically Significant and Repeatable: The standard tests are 

fixed and common for different users. They yield results that are 

the same, to a known level of precision. Unlike operational tests, 

they represent fair comparisons between systems, users, locations, 

and through time. 

Measurements do not replace Practice Scenarios: Coaches don’t 

just play games to select and train their players. They also use a 

variety of fitness tests and exercises, often standardized and 

replicable for anyone from a school student through to a 

professional. Similarly, Standard Test Methods focus the 

measurement of performance, help to determine capabilities and 

deficiencies and, along with Practice Scenarios, indicate when a 

system or pilot is ready for gametime and what position they are 

suited to play. 

21 
Evaluating Multiple Aspects of Sensing

Trade-offs reflect mission requirements

20 5

WHITE BUCKETS

ILLUMINATED WITH 

RED HEADLAMPS

Acuity and Field of View: Consider the need to see objects in detail 

and maintain situation awareness of the “big picture”. 

Motion, Latency, Radio Communications: Consider the effect of 

different resolutions on framerate, latency, and radio range. 

Light and Dark: Consider the need to observe detail in bright 

environments along with maintaining the ability to see in dim and 

dark conditions. 

Pointing and Zooming: Consider the need to pan and zoom to 

maintain situational awareness, observe what is needed in one 

visual frame, and maintain image stability. 

Live Interfaces vs Streamed vs Recovered:

Consider the difference in acuity when 

observed through the small interface

screen outdoors, compared to streamed

to a large monitor in a command vehicle, 

and compared to images from a memory 

card recovered from the drone. 

Visualizations Compare Tradeoffs: A coach 

doesn’t just look at the 40 ft dash when 

deciding on a player’s potential, so users 

should not just look at single tests. A whole 

suite of tests provides a more complete 

understanding of how a particular drone 

or pilot proficiency fits mission 

requirements and what 

the trade-offs are between capabilities. 

For instance, Visual Acuity and Field of 

View often, but not always, trade off. 

Latency

Video Latency: 

• Measures the time between an event being observed by the drone, and 

that event being shown on the operator screen. 

• A high speed recording camera (e.g. a cellphone in 120 frames per 

second high speed/slow motion video mode) is used to record a flashing 

light, seen both directly and through the drone via its operator screen, 

for 10 flashes. 

• The high speed video is inspected and the number of frames between 

the light turning on when observed directly, and when observed 

through the drone via its operator screen, divided by the high speed 

recording camera’s framerate, is the video latency, and averaged over 

the 10 flashes. 

Control Latency: 

• Measures the time between a control input being commanded and the 

drone moving based on that command. 

• A high speed recording camera is used to record both the operator 

control unit and the drone at the same time. 

• The operator quickly pushes the yaw control (generally the left stick) 

fully to one side to command a yaw and releases it once the robot 

begins to move. The operator repeats this 10 times. 

• The high speed video is inspected and the number of frames between 

the control being fully applied and the drone starting to move, divided 

by the high speed recording camera’s framerate, is the control latency, 

and averaged over 10 yaw commands. 

HOVERING DRONE

WITH CAMERA

VIEW FROM HIGH SPEED RECORDING CAMERA

SHOWING LIGHT DIRECTLY OBSERVED AND 

SEEN THROUGH OPERATOR INTERFACE

Drone 1

Drone 2

Drone 3

Same view, from the same drone 

and camera, at the same time. 

Interface Streamed Recovered
6 19 Positioning for Testing

Conducted at Chosen Hover Altitudes and at the 

Limits of Radio Communications

ALIGN BUCKETS AT THE CHOSEN HOVER ALTITUDE 

TO ENSURE CORRECT POSITIONING. 

Aligned. Perform test

while ensuring drone

maintains alignment. 

ALIGN WITH BUCKETS AND OBSERVE TARGETS

Not aligned. Move

until aligned before

performing test. 

Dual bucket alignments 

define High Hover altitudes of: 

• 90 m (300’). 

• 60 m (200’)

• 30 m (100’)

Triple bucket alignments 

define Hover altitudes of:

• 12 m (40’)

• 6 m (20’) 

SENSORS | POINTING and ZOOMING

Circle one per line. 

Lit Dark

Quick Complete

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Start time (clock image): ___:___:___

End time (clock image): ___:___:___

Total correct / Elapsed time

= Efficiency (rate) ________ rings/min

End time (clock image): ___:___:___

Total correct / Elapsed time

= Efficiency (rate) ________ rings/min

Circle one per line. 

Lit Dark

Quick Complete

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Start time (clock image): ___:___:___

Altitude is chosen either based on the mission requirements, or 

such that the system scores 2 to 4 rings into each Concentric C. 

If the system only scores 1 ring, move closer! 
7

• Buckets may be lit to assist in 

positioning during dim and dark 

testing conditions. Charts and targets 

should be lit from/by the drone. 

• An upward pointing center bucket, and one (for High Hover) or two 

(for Hover) angled bucket(s) point to the correct location above the 

charts to perform the test. 

• Robot shall maintain its position between observing the buckets 

and performing the test. Robot should look back at buckets 

periodically to check that it is still aligned during the test. 

• Tests at the limits of radio communications are performed with dual 

bucket alignments at four different orientations to test antenna 

directionality, with the operator station at increasing distances (and 

optionally within a vehicle), and with obstacles (buildings, ISO 

containers) between the operator station and drone. 

Pointing and Zooming

Visible, Thermal

• Test may be performed at Hover and High Hover altitudes 

(see p6), or in an Open Lane as shown above. 

• Two panels are placed under the hover location, one at a 

horizontal distance twice the hover height, one at a 

horizontal distance three times the hover height. 

• Each target has five Concentric C targets, as well as a 

rotating letter, hazmat label, license plate, operationally 

significant object, and thermal target. These relate 

measurements to operational capabilities. 

• Robot shall yaw 180 degrees between panels A and B 

(both under the robot).

• Repeat in daylight and in the dark, lit by the drone’s onboard lighting (if so equipped). 

Quick protocol: 

• Read as far in as possible into every Concentric Cs target 

on each panel before moving on to the next. 

Complete protocol: 

• Read the first Concentric C target on panel A, then panel B, 

C, and D, before returning to panel A to read the second 

Concentric C target, and so-on. 

18
TARGETS / MIN

TOTAL CORRECT

Teams Rotate Through Each Role

Each Pilot flies a 5-minute trial with help from others. 

Test in 3-4 person teams. 

Four person teams always have one person getting their aircraft 

ready to launch right after the previous lands.

Three person teams work too, but require some time between each 

rotation to prepare the next aircraft.

PILOT

• Maintain control of the aircraft.

• Call out each intention of movement before doing so.

• Call out each acuity target gap or measurement.

PROCTOR

• Fill in the form header.

• Read the test procedures to the Pilot.

• Confirm, record, and attest to scoring after the trial.

VISUAL OBSERVER (VO)

• Maintain sight with the aircraft and surroundings.

• Repeat the Pilot’s intention of movement to confirm.

• Call out corrections and warnings as necessary. 

8 17

TEAM ROTATIONS

ON DECK

GET DRONE READY TO FLY

VO PILOT

FACING AWAY FROM

THE TEST LANE

PROCTOR

SENSORS | MOTION PERCEPTION AND TRACKING

Circle one per line. 

Lit Dark

Wide Zoom

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Small motion perception and tracking: 

Circle targets whose rotation is correctly 

identified. Write “T” in targets successfully 

tracked. Cross out incorrectly identified targets. 

Start time (clock image): ___:___:___

End time (clock image): ___:___:___

Large motion tracking: 

For 20 transits of the flag (10 in each direction), circle if the flag was initially 

tracked, motion predicted while obscured, and re-acquired. 

Start time (clock image): ___:___:___

Initial tracking: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Motion predicted: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Re-acquired: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

End time (clock image): ___:___:___
9

Motion Perception and Tracking

Visible, Thermal

Small movement perception: • 10 randomly rotating disks, black and white, with two 

different sizes of contrasting squares that correspond in size to 

the two largest Concentric C gaps, are observed at different 

hover altitudes and provide statistically significant measures 

of motion perception. 

• Correctly report if each target is rotating, and the direction of 

rotation. Repeat separately for each camera, zoomed in and 

out if so equipped. 

• In daylight, difference in heating between black and white 

allow thermal cameras to be tested. 

Small movement tracking (for systems so equipped): • Demonstrate the tracking of the squares on the rotating disks 

while no (stationary) objects in the view are flagged as 

moving. 

Large movement tracking (for systems so equipped): • A 60 cm (2 ft) long, 15 cm (6 in) wide plastic flag, of a 

contrasting color to the background, is towed along a 

continuous rope loop (e.g. using a canine lure course winch), 

across the view of the drone, at a speed of 1.3 to 2.6 m/s (5 to 

10 km/h, 3 to 6 mph). The center 2.4 m (8 ft) is obscured. 

• Demonstrate tracking of the flag, and prediction of the 

movement of the flag through the obscured area. 

Specialized Tests

Survey Acuity: Automatically 

readable visual acuity targets, 

consisting of QR codes of different

sizes, are placed at regular intervals 

along a line (for a test lane) or placed on objects of 

interest (when embedded in a scenario). The drone 

flies along the line or through the scenario at 

different heights, speeds, and conditions, under pilot 

control or in autonomous mapping mode. The 

resulting images and/or video are analyzed and the 

readable QR codes of each size are counted. This 

reflects the amount of the environment that was 

observed at different resolutions. A resolution 

wedge on the chart allows the QR code reading 

system (and its threshold for code detection) to be 

calibrated. 

3D Vision: The Visual Acuity and Field of View

tests are performed with targets consisting of 

concentric rings of 7 half spheres, one in each of 

the 8 directions with one missing (equivalent to 

the gap in the Concentric C pattern). This is scaled 

up and/or performed at close range as necessary. 

Payload Delivery: Measure the ability to reliably 

deliver a payload, by placing or dropping. Test is 

conducted within an Open Lane. Drone takes 

off with payload and flies to a stable 

hover over bucket stand 3, aligning 

with buckets 3 and 1C. Drone 

then flies straight over bucket

stand 4, captures an alignment

image of bucket 4, then 

places or drops payload. 

Flag being towed 

along rope loop

Rotating disks

16
10 15 Acuity and Field of View SENSORS | HAZMAT

Visible, Thermal

Acuity: 

• 20 Concentric C targets, black-on-white and white-on-black, at 

different hover altitudes, provide statistically significant 

measures of acuity. 

• Correctly report the orientation of each 

visible target. Repeat separately for each 

camera, zoomed in and out if so equipped. 

• Performance will differ when test is conducted through the 

operator interface screen, streamed to a large monitor, or 

based on images recovered from a card on the drone. 

• In daylight, difference in heating between black 

and white allow thermal cameras to be tested.

Field of View: 

• At a chosen altitude, look straight down and use the markings, 

spaced at 1.2 m (4 ft), to determine the width of the field of 

view of the camera. 

• Repeat for each camera, zoomed in and out if so equipped. 

Circle one per line. 

Lit Dark

Wide Zoom

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Circle labels that are correctly identified. Cross out 

incorrectly identified labels. 

Start time (clock image): ___:___:___

End time (clock image): ___:___:___

LABELS / MIN

TOTAL CORRECT LABELS
11

Hazmat

Visible, Thermal Circle one per line. 

Lit Dark

Wide Zoom

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Field of View: ______________ m/ft

Start time (clock image): ___:___:___

SENSORS | VISUAL/THERMAL ACUITY and FIELD OF VIEW

End time (clock image): ___:___:___

Total correct / Elapsed time

= Efficiency (rate) ________ rings/min

End time (clock image): ___:___:___

Total correct / Elapsed time

= Efficiency (rate) ________ rings/min

• 20 standard 10 cm (4 in) hazmat labels on white and 

black backgrounds, at different hover altitudes, provide 

statistically significant measures of the ability to use 

detail and color to identify patterns and text. 

• Correctly report the identity of each label. Repeat 

separately for each camera and zoomed in and out. 

• Repeat in daylight and in the dark, lit by the drone’s onboard lighting (if so equipped). 

• In daylight, difference in heating between black and 

white allow thermal cameras to be tested. 

Circle one per line. 

Lit Dark

Wide Zoom

Visible Thermal

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Field of View: ______________ m/ft

Start time (clock image): ___:___:___

14
13

Color Acuity SENSORS | COLOR

Visible Circle one per line. 

Lit Dark

Wide Zoom

Interface Streamed Recovered

Camera name: _____________

Hover Height: ______________ m/ft

Circle circles that are correctly identified. Cross 

out incorrectly identified circles. 

Start time (clock image): ___:___:___

End time (clock image): ___:___:___

COLORS / MIN

• 20 colored circles on white and black backgrounds, at 

different hover altitudes, provide statistically significant 

measures of color identification and potential for 

confusion between colors. 

• Correctly report the identity of the color of each circle. 

Repeat separately for each camera and zoomed in and 

out. 

• Repeat in daylight and in the dark, lit by the drone’s onboard lighting (if so equipped). 

• Black and white backgrounds allow users to see if 

particular colors get “lost” in a dark or light environment. 

• Colored rings in other tests (e.g. Visual Acuity and 

Hazmat) evaluate the ability to perceive color when 

presented as a thin line, rather than as a solid block. 

12