Page 1 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Emergency Response Robots Project Report

Date: October 4-8, 2021

Event: Unmanned Tactical Applications Conference (UTAC) hosted by FlyMotion

Location: Guardian Centers responder training facility, Perry, GA

Emphasis: Validated 20 standard test methods for evaluating aerial drone capabilities and remote 

pilot proficiency along with their related scenarios with quantitative scoring tasks.

Project Lead: Adam Jacoff

Intelligent Systems Division Email: RobotTestMethods@nist.gov

National Institute of Standards and Technology Website: RobotTestMethods.nist.gov

U.S. Department of Commerce Office: 301-975-4235

Gaithersburg, MD 20899 USA Mobile: 301-704-2323

Summary..................................................................................................................................................................2

Site Overview ..........................................................................................................................................................3

OPEN TEST LANES ...................................................................................................................................................4

Open Scenario: Wide Area Search......................................................................................................................7

Open Scenario: Vehicle Identification................................................................................................................8

OBSTRUCTED TEST LANES ......................................................................................................................................9

Obstructed Scenario: Bus Hostage Situation....................................................................................................13

Obstructed Scenario: Vehicle Takedown .........................................................................................................16

Obstructed Scenario: Hazardous Tanker Truck Accident and Fire..................................................................17

Obstructed Scenario: Structure Exterior Inspection........................................................................................18

Obstructed Scenario: Flooded House and Collapsed Bridge ..........................................................................21

Obstructed Scenario: Night Operations House Surveillance and Search.......................................................22

CONFINED TEST LANES .........................................................................................................................................23

Confined Scenario: Collapsed Structure Search and Inspection ....................................................................25

Confined Scenario: Subway Terrorist Attack ...................................................................................................26

Legged Mobility and Search Tests.......................................................................................................................27

Sensor Test Lane.....................................................................................................................................................29
Page 2 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Summary

NIST’s Emergency Response Robots Project, led by Adam Jacoff, validated 20 standard test methods for 

evaluating aerial drone capabilities and remote pilot proficiency at the Unmanned Tactical Applications 

Conference. The event was hosted by FlyMotion at the Guardian Centers responder training facility in Perry, 

GA on October 4-8, 2021. The NIST team, which includes emergency responders and test facility managers, 

helped familiarize more than 120 participants from all over the world with our standard test methods and 

related training scenarios embedded with quantitative measures of performance.

These standard test methods have been developed by NIST with support from the Science and Technology 

Directorate of the U.S. Department of Homeland Security (DHS). They are being standardized through the 

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09).

Our process for developing, validating, and standardizing a growing suite of more than fifty test methods for 

ground, aerial, and aquatic systems, includes several types of events. We host requirements workshops 

with emergency responders, validation testing with robot manufacturers, competitions with robotics 

researchers, and standards committee meetings with all the above. We also conduct inter-laboratory 

experiments with other facilities that host our test methods for their own evaluations. For all these various 

communities, our standard test methods provide a tangible language of robot tasks that enable quantitative 

measures of robot capabilities and remote operator/pilot proficiency. They help guide innovation, measure 

progress, and highlight breakthroughs that everybody can understand and appreciate.

Here is a video pointer showing the NIST team with an overview of the Structure Collapse Scenario: 

Here are video pointers to TV News segments showing from WGXA and NBC/WMGT.

Figure 1) NIST team included the following (left to right) Max Delo, U.S. Marshals Service (Ret.); Andy 

Olesen, Canadian Emergency Response Robots Association; Alex Fraley, University of Maryland; Tony 

Galladora, Montgomery County, MD Police Dept.; Ann Virts, NIST;James Lantz, Maryland State Police Dept.; 

Adam Jacoff, NIST; Howie Stockhowe, Virginia Beach Fire Dept.; Parry Boogard, DHS FEMA Urban Search 

and Rescue Task Forces.
Page 3 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 Figure 2) A wide variety of drone sizes and capabilities were used by attendees, which is essential to fully 

validate the standard tests and scenarios.

Site Overview

Figure 3) The ongoing NIST station was set up for the 120 participating pilots to familiarize themselves with 

the spectrum of standard test methods with examples of related scenarios. There were 20 concurrent tests 

and scenarios for everybody to try throughout the week.

This event also included tactical scenarios demonstrated live with actors as perpetrators, victims, and 

emergency responders. They showed how drones and other robots can be incorporated into fast-paced and 

chaotic emergency response operations including simulated gun shots, fires, and smoke.

After each scenario demonstration the NIST team embedded standard test apparatuses to guide drone 

pilots through a sequence of essential tasks resulting in a quantitative score. These embedded test 

apparatuses enable performance of drones and pilots to be measured, compared, and tracked over time, 

which was never possible in such complex scenarios.

A variety of firefighting, public safety, and search & rescue scenarios were conducted:

GUARDIAN CENTERS, PERRY, GA

NIST CONFINED

VEHICLES

GUARDIAN CENTERS, PERRY, GA

STRUCTURE COLLAPSE AREA

SENSORS

OBSTRUCTED

STRUCTURE

OBSTRUCTED

TEST LANES

CONFINED TEST LANES

S1

S1

CONFINED

INTERIORS

L1

L2 L3 L4 L5 S1

S1

OBSTRUCTED

VEHICLE

L5 L4 L3 L2 L1

SEARCH

OPEN AREA

VEHICLE

L1 L2

L3

S1S2

(3) OPEN TESTS: 10/10 /20 FT LANES CONCURRENT

POSITIONS, TRAVERSE, ORBIT, INSPECT, RECON

(5) OBSTRUCTED TESTS: 5 LANES CONCURRENT

PERCH, WALL, GROUND, ALLEY, POST

(5) CONFINED TESTS: 5 LANES CONCURRENT

PERCH, WALL, GROUND, ALLEY, POST

(1) SENSOR ACUITY / CAMERA POINTING / SURVEY

NIST

CONFINED

OBSTRUCTED

OPEN

GUARDIAN CENTERS, PERRY, GA

STRUCTURE COLLAPSE AREA

E

A

S

Y

H

A

R

D

SENSORS

(1) SCENARIO: HIGH HOVER VEHICLE IDENTIFICATION

(1) SCENARIO: HIGH HOVER WIDE AREA SEARCH

(1) SCENARIO: EXTERIOR VEHICLE INSPECTION

(1) SCENARIO: EXTERIOR STRUCTURE SEARCH

(1) SCENARIO: INTERIOR VEHICLE INSPECTION

(1) SCENARIO: INTERIOR STRUCTURE SEARCH

1

2

3

4

5

6

6

4

5

3

2

1
Page 4 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

48

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

est Methods for Small Unmanned Aircraft Systems

national Standards Committee on Homeland Security Applications; 

onse Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

e emergency 

dous tasks from 

nal Institute of 

nternational effort 

p manufacturers, 

bjectively evaluate 

ficiency to align 

es the safety and 

as they save lives 

s

te pilots is to get 

ck. That’s where 

e. These test 

-5) and Payload 

ted across the 

ning with 

roficiency. They are 

ne can measure 

are their proficiency 

ar 

et up to enable 

valuate 

e ASTM 

Homeland Security 

54.09). They are also 

ments in 

Standard for Small 

blic Safety 

tandard Guide for 

f Unmanned 

8.03 F3266-18). 

rovide common 

ve results. They can 

 or embedded into 

able and 

nt qualitative 

tests set their own 

ot performance to 

nd mission 

sier to make and 

e performance data 

MAN 1-5) and Payload Functionality (PAY 1-5)

afety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS
Page 5 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 5) There are two ways to conduct test trials in the most basic Open Test Lane. LEFT: Basic 

Maneuvering trials use only white buckets to guide alignments with to capture NO ZOOM images. RIGHT: 

Payload Functionality trials add some operational workload to each bucket alignment task by capturing 

MAX ZOOM images of the visual/thermal acuity targets. They also use alternating white and black buckets 

to maximize exposure control.

Figure 6) Open test lanes with all white buckets are used to evaluate basic Maneuvering trials only, which is 

the typical starting point for novice pilots. Test lanes with alternating white and black buckets are used to 

evaluate operational readiness and system capabilities.

Conduct Tests Two Ways

Open Test Lane

Each lane uses (4) omnidirectional bucket stands, a 

Launch/Land Platform, and a measuring taps 

centerline. The parts for each stand include:

[1] 4x4x6in center post

[4] 2x4x12in legs with 45deg taper both ends

[5] 2-gallon white buckets (8in diameter)

10] 3in screws to affix the legs – 2 per leg at top

[5] 1in screws to affix the buckets

[4] Slotted leg extensions

[8] 1⁄4-20 x 2in hanger bolts and wing nuts for the

leg extensions to slide on and tighten

[15] 8in diameter weatherproof stickers

(see website to download the sticker files)

[4] Big numbers 1-1-1-1 inside the top bucket

[4] Big letters A-B-C-D around the top bucket

[5] Acuity targets A-B-C-D inside bottom of all

[2] Perch acuity targets inside and bottom of A

.

The bucket stands are adjusted to vertical using the slotted leg extensions so the angled buckets are at 45 degrees.

Stickers inside each bucket have a GREEN INSCRIBED RING 

to guide alignment and a visual acuity target with 

increasingly small Concentric Cs gaps to identify the 

correct (1 of 8) orientations.

TOP

( T )

( B )

BOTTOM 

( TR ) TOP RIGHT

( R ) RIGHT

( BR ) BOTTOM RIGHT

TOP LEFT ( TL )

LEFT ( L )

BOTTOM LEFT ( BL )

A

Gap size (mm) Gap size (inches) Equivalent Avg. Human vision at 6 m (20 !)

40 mm 1.58 in 4.5 %

16 mm 0.63 in 11 %

6.4 mm 0.25 in 28 %

2.6 mm 0.10 in 70 %

1.0 mm 0.04 in 180 %

0.41 mm 0.016 in 440 %

For accurate calibration, the outer diameter of the largest Landolt C optotype should be 200 mm (7.9 in). 

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; Operational Equipment; Robots (E54.08.01)

Nested Landolt C Vision Test Chart

#3

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

A

D

C

B 1, 2, 3, 4

LAUNCH

AD

PERCH 

ACUITY

P1

P1

LEAVE CARRY HANDLE ON TOP BUCKET

1A

1A 1A ALIGNED (5 PTS)

PARTIAL (1 PT)

Align with each bucket long enough to capture a 

single alignment image (NO ZOOM) showing the

green ring inside the bucket. Score 5 points for a 

continuous green ring or 1 point for a partial green 

ring. Similar scoring for accurate or partial landings.

Align with each bucket long enough to capture a 

single alignment image (NO ZOOM) of each bucket 

AND a single acuity image (MAX ZOOM) of each 

target. Score 1 point for each correct identification of 

the 5 increasingly small Concentric C gap directions.

Basic Maneuvering (MAN)

ALIGN WITH BUCKETS AND LAND ACCURATELY

Payload Functionality (PAY) 

ALIGN AND IDENTIFY ACUITY TARGETS

20 ALIGNMENTS TOTAL UP TO 100 POINTS 20 TARGETS TOTAL UP TO 100 POINTS

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Montgomery County Police Drone Training & Evaluation Facility

Safety | Capabilities | Proficiency

DOJ/DHS National Unmanned Aircraft Systems Program Evaluation, August 2020

I

CONFINED TEST LANE

GPS DENIED

and

DARK (<1 LUX)

DUAL BUCKET 

ALIGNMENTS

AT 1 M (3 FT) 

PROXIMITY TO 

OBSTACLES

OPEN TEST LANE

B

LENGTH: 30/60 M (100/200 FT)

E

OBSTRUCTED TEST LANE

DUAL BUCKET 

ALIGNMENTS

AT 2 M (6 FT) 

PROXIMITY TO 

OBSTACLES

LENGTH: 15 M (50 FT) LENGTH: 7.5 M (25 FT)

ACCURATE (5 PTS)

BLACK BUCKETS & COLOR RINGS 

EMBEDDED INTO SCENARIOS

1

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT

ALL “A” BUCKETS

POINT TOWARD THE 

LAUNCH/LAND

HANDLE

CENTERLINE

ONLY THE NUMBERS 

CHANGE FOR EACH 

OMNI BUCKET STAND

1A

BROKEN INNER BLACK EDGE

IS THE JUDGEMENT POINT

POST

LEGS 

FLUSH WITH TOP

SHOWN NOT ALIGNED SHOWN ALIGNED

WHITE BUCKETS & GREEN RINGS 

IN STANDARD TEST LANES
Page 6 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 7) The Open Test Lane scales to fit inside a gymnasium or other auditorium for small aircraft at low 

altitudes or outside for larger aircraft at higher altitudes.

Figure 8) The Open Test Lane uses the exact same set of apparatuses for 5 different tests.

The Open Test Lanes were located near related scenarios for wide area search of a plane crash site and a 

vehicle identification scenario. Two were set up with 3 m (10 ft) spacing between omni bucket stands and 

one was set up with 6 m (20 ft) spacing to be operationally relevant for various scenarios. All were set up 

with alternating white and black bucket stands to evaluate remote pilot proficiency using the Payload 

Functionality variant of the tests rather than the basic Maneuvering variant of the tests that use all white 

buckets throughout. The Payload Functionality tests involve a modest workload for the remote pilot to pantilt-zoom, focus, and control exposure at each bucket for scoring after the trial. The alternating white and 

black buckets force maximum exposure control. 

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

11

Scalable for Indoor/Outdoor Venues

Open Test Lane

MEASURING TAPE CENTERLINE 

UP TO 90 M (300 FT)

1

2

3

FLIGHT LINE

4

5(S)

S S

S

z S

ALTITUDE S

ALTITUDE 2(S)

S

ALTITUDE 5(S)

LAUNCH/LAND 

PLATFORM

OMNI BUCKET STANDS

(A-B-C-D)

SPACING = S ALTITUDES = S, 2(S), 5(S) LENGTH = 10(S) 

S = 3 m (10 ft) 3, 6, 15 m (10, 20, 50 ft) 30 m (100 ft) 

S = 6 m (20 ft) 6, 12, 30 m (20, 40, 100 ft) 60 m (200 ft) 

S = 9 m (30 ft) 9, 18, 45 m (30, 60, 150 ft) 90 m (300 ft)

Spacing (S) = Altitude

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

24 INCH 0R 48 INCH 

LAUNCH / LAND

PLATFORM

ACUITY TARGET

FOR ALIGNMENT

DIGITAL CLOCK CAPTURES

DATE | TIME (SYNCED)

WHITEBOARD CAPTURES

MAKE | MODEL | CONFIG

(4) OMNI BUCKET STANDS

12 INCH RADIUS

CIRCLE FOR ACCURACY

CENTERLINE 

MEASURING TAPE

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

1–4

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT “A” BUCKET

ALWAYS POINTING 

TOWARD THE 

LAUNCH/LAND

PLATFORM

HANDLE

CENTERLINE

LAUNCH / LAND PLATFORM

DIGITAL CLOCK

DATE & TIME

WHITEBOARD INFO

PILOT & AIRCRAFT

(4) OMNI 

BUCKET STANDS

CENTERLINE 

MEASURING TAPE

MEASURING TAPE CENTERLINE 

UP TO 90 M (300 FT)

1

2

3

FLIGHT LINE

4

5(S)

S S

S

z S

ALTITUDE S

ALTITUDE 2(S)

S

ALTITUDE 5(S)

LAUNCH/LAND 

PLATFORM

OMNI BUCKET STANDS(A-B-C-D)

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

11

Scalable for Indoor/Outdoor Venues

Open Test Lane

MEASURING TAPE CENTERLINE 

UP TO 90 M (300 FT)

1

2

3

FLIGHT LINE

4

5(S)

S S

S

z S

ALTITUDE S

ALTITUDE 2(S)

S

ALTITUDE 5(S)

LAUNCH/LAND 

PLATFORM

OMNI BUCKET STANDS

(A-B-C-D)

SPACING = S ALTITUDES = S, 2(S), 5(S) LENGTH = 10(S) 

S = 3 m (10 ft) 3, 6, 15 m (10, 20, 50 ft) 30 m (100 ft) 

S = 6 m (20 ft) 6, 12, 30 m (20, 40, 100 ft) 60 m (200 ft) 

S = 9 m (30 ft) 9, 18, 45 m (30, 60, 150 ft) 90 m (300 ft)

Spacing (S) = Altitude

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

24 INCH 0R 48 INCH 

LAUNCH / LAND

PLATFORM

ACUITY TARGET

FOR ALIGNMENT

DIGITAL CLOCK CAPTURES

DATE | TIME (SYNCED)

WHITEBOARD CAPTURES

MAKE | MODEL | CONFIG

(4) OMNI BUCKET STANDS

12 INCH RADIUS

CIRCLE FOR ACCURACY

CENTERLINE 

MEASURING TAPE

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

1–4

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT “A” BUCKET

ALWAYS POINTING 

TOWARD THE 

LAUNCH/LAND

PLATFORM

HANDLE

CENTERLINE

LAUNCH / LAND PLATFORM

DIGITAL CLOCK

DATE & TIME

WHITEBOARD INFO

PILOT & AIRCRAFT

(4) OMNI 

BUCKET STANDS

CENTERLINE 

MEASURING TAPE Spacing (S) = Altitude

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Lanes for Indoors and Outdoors

Open Test Lane

MEASURING TAPE CENTERLINE 

UP TO 90 M (300 FT)

1

2

3

FLIGHT LINE

4

5(S)

S S

S

z S

ALTITUDE S

ALTITUDE 2(S)

S

ALTITUDE 5(S)

LAUNCH/LAND 

PLATFORM

OMNI BUCKET STANDS

(A-B-C-D)

SPACING = S ALTITUDES = S, 2(S), 5(S) LENGTH = 10(S) 

S = 3 m (10 ft) 3, 6, 15 m (10, 20, 50 ft) 30 m (100 ft) 

S = 6 m (20 ft) 6, 12, 30 m (20, 40, 100 ft) 60 m (200 ft) 

S = 9 m (30 ft) 9, 18, 45 m (30, 60, 150 ft) 90 m (300 ft)

Altitudes = Ground Distance Between Stands

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

24 INCH 0R 48 INCH 

LAUNCH / LAND

PLATFORM

ACUITY TARGET

FOR ALIGNMENT

DIGITAL CLOCK CAPTURES

DATE | TIME (SYNCED)

WHITEBOARD CAPTURES

MAKE | MODEL | CONFIG

(4) OMNI BUCKET STANDS

12 INCH RADIUS

CIRCLE FOR ACCURACY

CENTERLINE 

MEASURING TAPE

NOT ALIGNED

1A

1A

1A

ALIGNED BARELY

ALIGNED WELL

BROKEN RING

PARTIAL RING

COMPLETE RING

COMPLETE RING

ALIGNED (5 PTS)

ALIGNED (5 PTS)

PARTIAL (1 PT)

5 Different Tests – Simulated and Physical

Open Test Lane

1

2

3

FLIGHT LINE

4

z

z

ALTITUDE S FLY STRAIGHT AND LEVEL OVER THE LANE CENTERLINE

START | END

ALIGN

NON-STOP

NON-STOP NON-STOP

NON-STOP

NON-STOP

NON-STOP

z

S

S S S S

5(S)

S

ALIGN CW

CCW

Recon

MAN/PAY 5

Spiral 

MAN/PAY 4

Orbit 

MAN/PAY 3

Traverse

MAN/PAY 2

Position

MAN/PAY 1

1

2

3

4

z

z

FREE FLIGHT

z z

z

CW

CCW

CW

CCW

LAND

S

S S S S

FLIGHT LINE

1

2

3

FLIGHT LINE

4

START | END

S

S S S S

5(S)

ALTITUDE S

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

18

1

2

3

FLIGHT LINE

4

ALTITUDE S

START | END

5(S)

S

S S S S

Traverse 

(MAN 2 | PAY 2)

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

14

Position 

(MAN 1 | PAY 1)

FLIGHT LINE

4

z

z

z

ALTITUDE S

START | END 3

2

1

5(S)

ALTITUDE 2(S)

S

S S S S

Inspect
Page 7 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 Figure 9) LEFT: An Open Test Lane with 3 m (10 ft) spacing and alternating white and black omni bucket 

stands for the Payload Functionality variant of the tests. RIGHT: Multiple test lanes set up to be managed by 

a single Proctor helping teams of pilots and their visual observers learn to fly the paths and fill out the forms 

correctly. RIGHT: A 6 m (20 ft) lane spacing set up to be more relevant for some scenarios.

Open Scenario: Wide Area Search

Quad Screen Trial Video Example - Wide Area Search

The Open Test Lane evaluates flight paths to identify objects from safe altitudes in open environments. 

Related scenarios include wide area searches such as this simulated plane crash site that used all the same 

omni bucket stands from the Open Test Lane. There are 20 targets overall, each with 5 increasingly small 

features to identify for a total of up to 100 points available for a complete trial. This enables comparison of 

scores for pilots and aircraft that can reliably perform the various bucket alignments and identify the smallest 

visual/thermal acuity features across all available acuity targets. The trial time limit is typically set to 20 

minutes to remain within one battery charge and to maintain a schedule throughout the day for multiple 

pilots. Time limited trials also enable direct comparisons of scores for completeness and efficiency. Only 

scores using similar aircraft and trial times are directly comparable to evaluate pilot proficiency, but a variety 

of different aircraft can be used to compare overall scores and ease of use.

 

Figure 10) LEFT: The wide area search was this simulated plane crash site across several acres. RIGHT: There 

was also a separate vehicle identification scenario on a dirt road near trees.

WHITE BUCKET ALIGNMENTS

BLACK BUCKET ALIGNMENTS
Page 8 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 11) Quad screen trial videos capture the following concurrent views. TOP LEFT: The pilot’s interface 

showing the thermal identification of a hand warmer inside an black omni bucket stand to represent a 

partially exposed survivor within the wreckage. BOTTOM LEFT: All operator inputs to the system. TOP 

RIGHT: An overview of the scenario. BOTTOM RIGHT: A detailed view of the drone in proximity to the 

objects of interest.

Open Scenario: Vehicle Identification

Quad Screen Trial Video Example - Vehicle Identification

(Scenario layout training: find the errors in the sticker placements around the vehicle)

The Open Test Lane also leads pilots toward vehicle identification scenarios from safe hover altitudes above 

nearby obstacles. The designated flight path around the vehicle includes an orbit with equal radius and 

altitude to align with the 45-degree angled omni buckets on the roof (bucket targets A1, B1, C1, D1). The 

chosen orbit radius and altitude is based on the height of surrounding obstacles, the intended mission 

requirements, the aircraft’s zoom capabilities, etc. Any orbit can be used, but only trials with similar orbits 

and trial times are comparable. There are also precise perch positions and orientations on the road directly 

under the front and rear orbit positions. These perches evaluate landing accuracy along with the functional 

pan-tilt-zoom capabilities of the aircraft while landed at the chosen orbit radius. Perching demonstrates the 

aircraft’s capability to maintain surveillance while conserving battery. The perch targets are buckets under 

the vehicle that represent operationally significant underbody objects in shadow (bucket targets A5, C5). 

Each side of the vehicle has 5 exterior visual/thermal targets to identify from the chosen radius, altitude, 

and perch positions. There are 20 targets overall, each with 5 increasingly small features to identify for a 

total of up to 100 points available for a complete trial. All targets are identified from the designated flight 

path starting with the angled buckets on the roof to verify the altitude and radius, then exterior features or 

surrounding ground objects, and exterior window targets to identify while presumably looking for interior 
Page 9 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

objects. All targets are on the exterior of the vehicle to eliminate potential variations in scoring due to

interior target obstructions, window glare, tinting, etc. Interior target identifications can be considered 

bonus points because they are less comparable across trials due to changes in sunlight. 

 

Figure 12) LEFT: The Open scenario for vehicle identification includes a flight path with a chosen orbit radius, 

altitude, and perch positions based on the height of surrounding obstacles, the intended mission 

requirements, the aircraft’s zoom capability, etc. Any orbit can be used, but only trials with similar orbits and 

trial times are comparable. RIGHT: The recipe for embedding 20 visual/thermal acuity targets.

OBSTRUCTED TEST LANES

The Obstructed Test Lanes enable remote pilots to fly safe and repeatable flight paths to inspect objects 

within close proximity to obstructions. They include a comprehensive set of 5 different tests that guide 

remote pilots through various standoff positions, orientations, and perches at 2-3 m (6-10 ft) from objects. 

They can be performed outdoors or indoors to control lighting, weather, and access to the Global Positioning 

System (GPS).

 Figure 13) LEFT: The Obstructed Test Lanes are typically set up in order of increasing difficulty left to right as 

Perch, Wall, Ground, Alley, and Post. RIGHT: The Post test shown requires circumnavigating all the way 

A1 – FRONT SIDE OMNI B1 – PASSENGER SIDE OMNI

D1 – DRIVER’S SIDE OMNI C1 – REAR OMNI

REAR PERCH

FRONT PERCH

ROOF OMNI BUCKET TARGETS

(A1, B1, C1, D1) 

5 TARGETS TO IDENTIFY ON EACH SIDE = 20 TARGETS = 100 POINTS

STICKERS ON ALL SIDES OF THE VEHICLE 

AND SURROUNDING GROUND

PERCH BUCKETS TARGETS

FRONT AND REAR

1 A1 – FRONT SIDE – ROOFTOP OMNI BUCKET 

2 A2 – FRONT SIDE – WINDSHIELD CENTER FOR INTERIOR OBJECTS 

3 A3 – FRONT SIDE – VIN # 

4 A4 – FRONT SIDE – LICENSE PLATE 

5 A5 – FRONT SIDE – PERCH POSITION UNDERBODY BUCKET 

6 B1 – PASSENGER SIDE – ROOFTOP OMNI BUCKET 

7 B2 – PASSENGER SIDE – FRONT WINDOW FOR INTERIOR OBJECTS 

8 B3 – PASSENGER SIDE – REAR WINDOW FOR INTERIOR OBJECTS 

9 B4 – PASSENGER SIDE – EXTERIOR FEATURE OR SURROUNDING GROUND 

10 B5 – PASSENGER SIDE – EXTERIOR FEATURE OR SURROUNDING GROUND 

11 C1 – REAR SIDE – ROOFTOP OMNI BUCKET 

12 C2 – REAR SIDE – WINDOW CENTER FOR INTERIOR OBJECTS 

13 C3 – REAR SIDE – LICENSE PLATE 

14 C4 – REAR SIDE – EXTERIOR FEATURE OR SURROUNDING GROUND 

15 C5 – REAR SIDE – PERCH POSITION UNDERBODY BUCKET 

16 D1 – DRIVER SIDE – ROOFTOP OMNI BUCKET 

17 D2 – DRIVER SIDE – FRONT WINDOW FOR INTERIOR OBJECTS 

18 D3 – DRIVER SIDE – REAR WINDOW FOR INTERIOR OBJECTS 

19 D4 – DRIVER SIDE – EXTERIOR FEATURE OR SURROUNDING GROUND 

20 D5 – DRIVER SIDE – EXTERIOR FEATURE OR SURROUNDING GROUND 

PERCH WALL GROUND ALLEY

POST
Page 10 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

around to align with high and low buckets while passing between the post and the wall as if inspecting a 

vehicle parked along a 3 m (12 ft) curb near a building. 

PERCH TEST

The drone lands 2 m (6 ft) from the wall 

guided by dual bucket alignments in various 

orientations:

A single perch measures the field of view of 

an independent pan-tilt-zoom camera. 

Others may need to re-launch, rotate, and 

land to identify all the targets in order.

WALL TEST

The drone flies within 2-3 m (6-10 ft) of the

wall at 45 degrees from forward of the 

aircraft to align with buckets and identify

features upward and downward. This 

approximates inspection tasks inside

elevated windows and doors.

GROUND TEST

The drone flies within 2-3 m (6-10 ft) of the

ground to align with buckets and identify

features leftward and rightward. This 

approximates inspection tasks around ground 

objects.

ALLEY TEST

The drone flies within 2-3 m (6-10 ft) of the 

wall and ground along a line parallel to the 

wall while rotating in place to identify 

features all around. This approximates

inspecting ground objects near walls or other 

obstacles. 

POST TEST

The drone flies within 2-3 m (6-10 ft) all 

around the post at different elevations to 

inspect objects upward and downward. This 

approximates inspecting elevated objects 

near walls or other obstacles.

Page 1

Obstructed Test Lane

 of motion, align with as many buckets as possible. Pilot proficiency is only compared using similar systems. 

(6 ft) of the 

rst landing 

ependent 

 of the wall or 

of the aircraft 

downward.

 of the wall or 

aft (90 

ward and 

of the wall or 

rees) and 

le inspecting 

 of the post 

bstacle and 

nd downward.

afe and repeatable positions within close proximity to a wall or other large 

pection tasks. Each of the 5 tests requires flying in different orientations to 

ircraft facing the object and gradually turning the drone’s back to the 

guide the pilot to within roughly 2 m (6 ft) when perfectly aligned with both 

understand the look and feel of flying close to objects. The same Dual 

milarly within operational training scenarios. 

Version: 2020B6b Page 1

Obstructed Test Lane

 of motion, align with as many buckets as possible. Pilot proficiency is only compared using similar systems. 

(6 ft) of the 

rst landing 

ependent 

 of the wall or 

 of the aircraft 

downward.

 of the wall or 

aft (90 

tward and 

 of the wall or 

grees) and 

le inspecting 

 of the post 

bstacle and 

nd downward.

afe and repeatable positions within close proximity to a wall or other large 

pection tasks. Each of the 5 tests requires flying in different orientations to 

ircraft facing the object and gradually turning the drone’s back to the 

guide the pilot to within roughly 2 m (6 ft) when perfectly aligned with both 

understand the look and feel of flying close to objects. The same Dual 

milarly within operational training scenarios. 

Version: 2020B6b Page 1

Obstructed Test Lane

 of motion, align with as many buckets as possible. Pilot proficiency is only compared using similar systems. 

(6 ft) of the 

rst landing 

ependent 

 of the wall or 

of the aircraft 

downward.

 of the wall or 

ft (90 

ward and 

of the wall or 

rees) and 

le inspecting 

 of the post 

bstacle and 

nd downward.

afe and repeatable positions within close proximity to a wall or other large 

pection tasks. Each of the 5 tests requires flying in different orientations to 

ircraft facing the object and gradually turning the drone’s back to the 

uide the pilot to within roughly 2 m (6 ft) when perfectly aligned with both 

understand the look and feel of flying close to objects. The same Dual 

milarly within operational training scenarios. 

nse Robots (E54.09) | Website: RobotTestMethods.nist.gov

Version: 2020B6b Page 1

Obstructed Test Lane

 of motion, align with as many buckets as possible. Pilot proficiency is only compared using similar systems. 

6 ft) of the 

rst landing 

ependent 

 of the wall or 

of the aircraft 

downward.

 of the wall or 

ft (90 

ward and 

of the wall or 

rees) and 

e inspecting 

 of the post 

bstacle and 

nd downward.

fe and repeatable positions within close proximity to a wall or other large 

pection tasks. Each of the 5 tests requires flying in different orientations to 

ircraft facing the object and gradually turning the drone’s back to the 

uide the pilot to within roughly 2 m (6 ft) when perfectly aligned with both 

understand the look and feel of flying close to objects. The same Dual 

milarly within operational training scenarios. 

est Methods for Small Unmanned Aircraft Systems

ational Standards Committee on Homeland Security Applications; 

nse Robots (E54.09) | Website: RobotTestMethods.nist.gov

Version: 2020B6b Page 1

Obstructed Test Lane

e of motion, align with as many buckets as possible. Pilot proficiency is only compared using similar systems. 

(6 ft) of the 

rst landing 

ependent 

 of the wall or 

 of the aircraft 

 downward.

 of the wall or 

aft (90 

tward and 

 of the wall or 

grees) and 

le inspecting 

 of the post 

bstacle and 

nd downward.

afe and repeatable positions within close proximity to a wall or other large 

pection tasks. Each of the 5 tests requires flying in different orientations to 

ircraft facing the object and gradually turning the drone’s back to the 

guide the pilot to within roughly 2 m (6 ft) when perfectly aligned with both 

understand the look and feel of flying close to objects. The same Dual 

milarly within operational training scenarios. 
Page 11 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

The Obstructed test apparatuses use “dual bucket alignments” to enable remote pilots to triangulate a safe 

standoff position by simultaneously aligning with a perpendicular bucket and its associated angled 

bucket. The dual bucket alignments form a right triangle with equal dimensions for the bucket separation 

and the aircraft standoff. They use pairs of 7.5 liter (2 gallon) buckets with 20 cm (8 in) diameter recessed 

targets inside at a 2 m (6 ft) spacing.

Alignment rings inside the perpendicular (90 degree) white buckets visually guide safe flight paths toward 

and away from the objects. Alignment rings inside the associated angled (45 degree) buckets indicate when 

the aircraft is in the designated safe and repeatable standoff position. That’s where a remote pilot can 

maintain position and operate their interface to identify increasingly small features on acuity targets inside 

the buckets. This modest operational workload includes zoom, focus, and exposure to capture images for 

scoring after the trial. White and black buckets require maximum control of exposure levels or thermal 

palettes to discern more details and score more points.

There are 5 different tests with increasing difficulty called Perch, Wall, Ground, Alley, and Post. The dual 

bucket alignments guide remote pilots through a series of 10 positions, orientations, and perches within 

both the standard test lanes and the operational scenarios embedded with scoring tasks. All tests and 

scenarios result in quantitative scores up to 100 points maximum to facilitate measurement, tracking, and 

comparison across different aircraft and/or remote pilots.

Procedure: The procedure is the same for all Obstructed tests. Each test has dual bucket alignments 

designated 1 – 4 that at are performed in a sequence. The sequence includes some backtracking to ensure 

the tasks are performed in various directions relative to the obstacles involved. The sequence of positions is 

1 2 3 4 – 3 2 1 – 2 3 4 with the red underlined numbers indicating the backtracking part of the sequence. 

That results in 10 dual bucket alignments or 20 bucket alignments total.

Maneuvering Trials: A complete trial totals up to 100 points maximum for 20 bucket alignments. Points are 

scored using a single no zoom image of each bucket showing either a full alignment ring (5 points), a partial 

alignment ring (1 point), or no alignment ring (0 point). 

Acuity Targets: All trials result in maneuvering scores. But Payload Functionality trials add an operational 

workload to identify acuity targets while aligned with buckets. The level of detail the system can discern 

should be known and set prior to conducting a Payload Functionality trial. Each acuity target has 5 

increasingly small gap orientations to identify correctly. The smallest features are 1 mm (0.04 in) needed to 

read small text on shipping labels, for example. Each identifiably gap orientation is verbally conveyed to a 

Proctor during the trial, or when operating alone a single full zoom image of the acuity target can be 

captured to score after the trial.

Time Limits: Test trials are not intended to be races. But trial time limits can be imposed to minimize fatigue 

across multiple tests or to maintain a schedule. Trial time limits should be long enough for an “expert” to 

complete a perfect trial. Scores of incomplete trials due to expired time limits can only be compared to 

trials with the same elapsed time limits. Typical time limits are typically set to 5 minutes for Maneuvering 

trials and 10 minutes for Payload Functionality trials. Although any time limits may be used depending on 

the drone, the pilot, and the environment.

Faults: A fault is any contact with a test apparatus or any safety issue such as crossing a designated 

boundary between test lanes or the remote pilot flight line. Any fault results in an end of trial.
Page 12 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Metrics: A complete trial requires performing all the designated bucket alignments in order with no faults 

(contact with an apparatus) or safety issues (exceeding a boundary). If the trial is not complete, the metrics 

below do not apply. Keep practicing until complete trials are routinely achieved before applying these 

metrics:

• Score (total points) – primary: Presuming a complete Maneuvering trial or Payload Functionality 

trial, the total points are a measure of the combined effectiveness of the aircraft system and the

remote pilot to maneuver through all the positions, orientations, and perches necessary to align 

with all the buckets. Trial scores add up to 100 points maximum. These scores can be used to 

compare remote pilot proficiencies when using the same drone and interface in the same test 

method. They also can be compared to the score of an “expert” pilot, typically provided by the 

manufacturer, which is considered the 100th percentile of proficiency on a particular aircraft 

system.

• Efficiency (elapsed time) – optional: Presuming a perfect score for a Maneuvering Trial or a Payload 

Functionality trial, the elapsed time is a measure of the combined efficiency of the aircraft system 

and the remote pilot. Elapsed trial times can help distinguish between perfect scores to identify 

more efficient techniques or approaches. 

Note: The verge points of the dual bucket alignments designate the most efficient locations to score all 

buckets. However, stable hovers at the verge points between buckets can be difficult to enforce similarly for 

various drone sizes and pilot proficiencies. So each bucket can be scored individually from any desired 

proximity, understanding that the resulting scores and trial times may be negatively affected.
Page 13 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Obstructed Scenario: Bus Hostage Situation

Quad Screen Trial Video Example - Bus Hostage Situation

The bus hostage scenario included role players as perpetrators and victims with actual law enforcement 

organizations from the region demonstrating simulated tactics including drones and a ground robot. 

 Figure 14) LEFT: One drone dropped a mobile phone to establish communications while another surveilled 

the situation. RIGHT: The ground robot also supported interactions prior to tactical intervention by law 

enforcement officers.

We added quantitative measures of performance to this scenario using dual bucket alignments to guide 

remote pilots safely through a designated series of inspection tasks. This enables comparison of scores up to 

100 points for pilots and aircraft that can reliably perform the various bucket alignments and identify the 

smallest visual/thermal acuity features across all available targets. The trial time limit was set to 20 minutes 

to remain within one battery charge and to maintain a schedule for multiple pilots. Time limited trials also 

enable direct comparisons of scores for completeness and efficiency. Only scores using similar aircraft and 

time times are directly comparable to evaluate pilot proficiency, but a variety of different aircraft can be used 

to compare overall scores and ease of use.

The flight paths included safe and repeatable standoff positions, orientations, and perches within 2 m (6 ft) of 

objects for aircraft flying the most efficient flight path. But bucket alignments can be achieved from any standoff 

for larger aircraft using more capable zoom lenses. The embedded scoring tasks are 7 liter (2 gallon) buckets 

with 20 cm (8 in) diameter recessed targets inside. The perpendicular (90 degree) white buckets provide 

visual alignments for the remote pilot to trust as safe vectors to approach and leave the object being 

inspected. The associated angled (45 degree) buckets enable triangulation to maintain a safe proximity 

from the object while operating their zoom, focus, and exposure to capture images for scoring after the 

trial.

Each set of tasks included 10 positions and orientations worth 10 points each distributed throughout the 

scenario. The pilot scores 5 points for getting aligned with each perpendicular bucket plus another 1-5 

points for correctly identifying increasingly small features on the acuity target located inside the associated

angled bucket. The smallest features to identify are 1 mm (0.04 in) representing small text on shipping 

labels, for example. The white and black pairs of buckets require maximum exposure control to capture 

images for scoring after the trial.
Page 14 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 15) There were two orbits to perform sequentially. The large dual bucket alignments defined

positions #1-10 to assess the situation quickly during the initial assault. Then the smaller dual bucket

alignments defined positions #11-20 to identify key details and gather evidence. Scenarios include an 

accurate perch position and orientation to demonstrate a persistent point of view that’s considered 

operationally important. In this case the large bucket #10 required landing with a view through the door up 

the stairs to the driver. Another exterior small bucket perch was set up for a suspicious package placed 

under the vehicle. There was also an interior perch location to identify a package found inside on the floor 

near a seat.

 

Figure 16) LEFT TO RIGHT: Exterior alignment with bucket 9 and 9A (white) look inside the door directly 

above the perch position. The aircraft perched while aligned with buckets 10 and 10A (black). The view from 

the aircraft while perched is a persistent view of the door and driver. Exterior perch positions are also 

designated for underbody inspection tasks using the smaller buckets during a second phase.

PERCH

WHITE BUCKET ALIGNMENTS BLACK BUCKET ALIGNMENTS
Page 15 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 17) The quad screen video of the bus hostage scenario showing the drone point of view in the perch 

position (upper left).

Large vehicles like buses can require interior searches as well. These relate to the Confined Test Lanes 

described later, but are typically part of these types of scenarios. They are intended only for scoring by 

aircraft that can safely fly inside the bus.

 

Figure 18) LEFT: An interior dual bucket alignment (buckets 1 and 1A) just inside the entry door at the 

driver’s seat. The perpendicular (90 degree) bucket provides a safe vector to approach the apparatus until 

the angled bucket (45 degree) is simultaneously aligned. That’s the safe 1m (3ft) standoff position at which 

the pilot should maintain position knowing the drone is safe from obstacles and work the interface controls 

for zoom, focus, and exposure to capture an image that can be scored after the trial. RIGHT: Other dual 

bucket alignments were related to views of objects on the seats, under the seats, and near the back door on 

the floor.
Page 16 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 Figure 19) LEFT: Other interior tasks using the small buckets in confined search spaces. CENTER: Two 

apparatuses require vertical alignments. RIGHT: Three apparatuses require horizontal alignments. This is the 

final perch position to maintain a view of an object of interest.

Obstructed Scenario: Vehicle Takedown 

Quad Screen Trial Video Example - Vehicle Takedown

Figure 20) LEFT: Vehicles of all sizes and conditions can be bucketized similarly for quantitative scoring, 

including panel vans and trucks. RIGHT: Dual bucket alignments guide the drone into position to look inside 

the front windshield from two different angles while using triangulation between the buckets to stay safe.

 

Figure 21) The bucket numbering leads the pilot around an orbit starting in front of the vehicle, then around 

to the driver side, and so on. The large buckets targets are identified in the first orbit to assess the situation 

WHITE BUCKET ALIGNMENTS

BLACK BUCKET ALIGNMENTS
Page 17 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

for up to 100 points. The small buckets targets are identified in the second orbit as evidence gathering for 

another 100 points, including where the gun landed during the demonstration.

Figure 22) The quad screen video of the tactical vehicle takedown with evidence gathering.

Obstructed Scenario: Hazardous Tanker Truck Accident and Fire

Quad Screen Trial Video Example - Hazardous Tanker Fire

The tanker truck accident and fire scenario used 5 large bucket apparatuses to guide remote pilots to points

of view around the vehicles for a fast tempo initial assessment. We also embedded 5 small bucket 

apparatuses near features needing detailed inspections such as the inside of the cab, dripping valves, and 

gauges to read. This scenario included drop accuracy tasks to emplace remote sensors trying to determine 

the nature of the hazard (see the sensors on the ground).

 

Figure 23) LEFT: The tanker accident scenario included a simulated fire and potential hazmat spill requiring 

firefighters to react as appropriate based on reconnaissance from the drones and dropped sensors. RIGHT: 

Thermal overview image from the drone and the dropped sensor packages.
Page 18 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 24) Our quad screen video captures simultaneous views of what the remote pilot sees through the 

interface, all interactions with the interface, and off-board views of the situation around the aircraft.

Obstructed Scenario: Structure Exterior Inspection 

Quad Screen Trial Video Example - Structure Exterior

Figure 25) The collapsed structure had several different scenarios to fly. The Obstructed scenarios (some 

shown with ovals) were perimeter windows, doors, and surrounding ground objects. The Confined scenarios 

(some shown with rectangles) were vehicles in the semi-collapsed parking garage and room-to-room 

searches inside the semi-collapsed building.
Page 19 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Structure exterior inspection tasks include looking through windows and doors along with surrounding 

ground objects of interest. In this case, the tasks were embedded around a partially collapsed 

structure. The objective was to safely fly in close proximity of about 2 m (6 ft) from the windows and doors 

to perform a window/door clearing maneuver with high/low and left/right views inside the structure. 

 

Figure 26) Obstructed apparatuses are set up in sets of vertical and horizontal tasks throughout the 

scenario.

Figure 27) The scenario façade was embedded with window inspection tasks on two sides with partially 

collapsed walls. Each Dual Bucket Alignment guides the remote pilot to positions and orientations with 

high/low and left/right views inside the windows.

ALIGNED DRONE LOOKING 

THROUGH THE WINDOW

VERTICAL

HORIZONTAL

ALIGNMENT TASKS IN 

WINDOW/DOORS
Page 20 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 28) Our quad screen video captures simultaneous views of a trial, including what the remote pilot 

sees through the interface, all interactions with the interface, and off-board views showing the situation 

around the aircraft. 

Figure 29) This final vertical test apparatus leads the remote pilot to perch with a view through the doorway. 

This was the 10th and final position and orientation in the scenario that totaled up to 100 points for a 

complete search through the windows of the façade on two sides of the structure.

PERCHED

DRONE

(ALIGNED)

WHITE BUCKET ALIGNMENTS

BLACK BUCKET ALIGNMENTS
Page 21 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Obstructed Scenario: Flooded House and Collapsed Bridge 

Quad Screen Trial Video Example - Flooded House Scenario

Several houses were submerged with victims on the roof and others in the water nearby. The tasks to 

perform included delivery of personal floatation devices to those stranded on the roof, searching in 

windows for survivors, and identifying/tracking moving swimmers. The omni bucket apparatuses replaced 

the actors as objects to identify and the landing served as target for the dropped payload accuracy. 

 

Figure 30) Victims on the rooftop of a flooded house were replaced with omni bucket stands to perform 

identification tasks and a landing was used as an accuracy target for dropping a personal floatation device.

 

Figure 31) LEFT: Other inspection tasks required the drone to hover low above the water to look in windows 

and doors. The two positions and orientations designated by the white and black pairs of buckets guided

the pilot to completely inspect the floating mattress looking for survivors (visual or thermal). RIGHT: Similar 

window inspection tasks were located under a more difficult covered porch near a door. That’s where 

underwater versions of these same tests were embedded for underwater system reconaissance.
Page 22 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 

Figure 32) A floating omni bucket task was pulled through the water by a remotely controlled rescue device

to represent floating or swimming victims. It provided a moving object for the drone to identify on all sides.

 

Figure 33) The flashflood bridge collapse included an overturned school bus and vehicles requiring a 

comprehensive search for survivors.

Obstructed Scenario: Night Operations House Surveillance and Search

This scenario was actually conducted the week prior at a different event, but nicely augments all of the 

scenarios above. This house surveillance at night used two sets of horizontal and vertical Obstructed test 

apparatuses on all four sides of the house, guiding the remote drone pilot to safe locations among a variety 

of very difficult obstacles such as the overhanging roof, trees, shrubs, power lines, shrubs, etc. A set of 5 

vertical test apparatuses with 10 positions and orientations were attached to all sides of the house totaling

100 points. A set of and 5 horizontal test apparatuses with 10 positions and orientations were also placed 

on or near objects of interest around the house for another 100 points. 
Page 23 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 34). Obstructed test apparatuses embedded into an exterior house search at night show the 

designated hover positions and orientations including a perch position in view of the back door. The aircraft 

was outfitted with lights to illuminate the scene but also used its thermal vision capabilities to negotiate the 

myriad of obstacles such as trees, power lines, shrubs, etc.

CONFINED TEST LANES

The Confined Test Lanes enable remote pilots to fly safe and repeatable flight paths to inspect objects 

within confined environments and interior room-to-room searches. They include a comprehensive set of 5 

different tests that guide remote pilots through various standoff positions, orientations, and perches at 1 m (3

ft) from objects. They are half the size of the Obstructed Test Lanes but use all the same procedures and scoring. 

They can be performed outdoors or indoors to control lighting, weather, and access to the Global Positioning 

System (GPS).

The Confined Test Lanes and related scenarios use “dual bucket alignments” to enable remote pilots to 

triangulate a safe standoff position by simultaneously aligning with a perpendicular bucket and its 

associated angled bucket. The dual bucket alignments form a right triangle with equal dimensions for the 

bucket separation and the aircraft standoff. They use pairs of 1 liter (1 quart) buckets with 10 cm (4 in) 

diameter recessed targets inside at a 1 m (3 ft) spacing.

PERCH
Page 24 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 Figure 35) The Confined Test Lanes are half the size of the Obstructed Test Lane but use all the same 

procedures and scoring. The are typically placed near related scenarios such as this semi-collapsed building 

and parking decks with interior room-to-room searches and other confined space inspections. LEFT TO 

RIGHT: Perch, Wall, Ground, Alley, and Post tests evaluate positions, orientations and perches within 1 m (3 

ft) of objects to identify features. 

Alignment rings inside the perpendicular (90 degree) white buckets visually guide remote pilots along safe 

flight paths toward and away from the objects being inspected. Alignment rings inside the associated angled 

(45 degree) buckets indicate when the aircraft is in the designated safe and repeatable standoff position. 

That’s where a remote pilot can maintain position and operate their interface to identify increasingly small 

features on acuity targets inside the buckets. This modest operational workload includes zoom, focus, and 

exposure to capture images for scoring after the trial. White and black bucket pairs require maximum 

control of exposure levels or thermal palettes to discern more details and score more points.

There are 5 different tests with increasing difficulty called Perch, Wall, Ground, Alley, and Post. The dual 

bucket alignments guide remote pilots through a series of 10 positions, orientations, and perches within 

both the standard test lanes and the operational scenarios embedded with scoring tasks. All tests and 

scenarios result in quantitative scores up to 100 points maximum to facilitate measurement, tracking, and 

comparison across different aircraft and/or remote pilots.

Scoring: Each dual bucket alignment is worth up to 10 points, including 5 points for maneuvering and 5 

points for visual/thermal acuity. Maneuvering points are scored using a single no zoom image of each 

perpendicular bucket showing either a full alignment ring (5 points), a partial alignment ring (1 point), or no 

alignment ring (0 point). Visual/thermal acuity points are scored using targets inside the angled buckets 

with 5 increasingly small gap orientations to identify correctly (1 point each up to 5 points). The smallest 

features to identify are 1 mm (0.04 in) representing small text on shipping labels, for example. Target 

identifications can be verbally conveyed to a Proctor during the trial, or a single full zoom image of each 

acuity target can be used to score the trial after landing. A complete trial includes 20 bucket 

alignments totaling up to 100 points maximum per trial.

Note: The dual bucket alignments define the most efficient positions to score all buckets placed around the 

object being inspected. However, stable hovers at the verge points between buckets can be difficult to 

enforce similarly for various aircraft sizes and pilot proficiencies. So each bucket can be scored individually 

from any desired proximity, understanding that the resulting scores and trial times may be negatively 

affected.

 PERCH WALL GROUND ALLEY POST
Page 25 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Confined Scenario: Collapsed Structure Search and Inspection

One of the confined scenarios at this event were semi-collapsed structures representing environments not 

safe for responders to search given the potential for further collapse due to aftershock or structural failures. 

The inspection tasks were placed around crushed vehicles within a collapsed structure. They were also

distributed across individual rooms in a search pattern to ensure each room was inspected for victims and

other objects of interest. 

Figure 36) The collapsed structure scenario at the Guardian Centers supported a wide variety of embedded 

test methods inside the structure and around objects of interest in semi-collapsed areas.

Figure 37) LEFT: Confined test apparatuses embedded into the semi-collapsed parking garage around 

vehicles that needed to be searched for survivors within a very complex three-dimensional environment. 

We placed 5 apparatuses around the vehicles to designate 10 positions and orientations to inspect. The 

white and black bucket pairs total up to 100 points for a complete trial. RIGHT: Confined test apparatuses 

embedded in a room-to-room search within the collapsed structure. These measured the completeness of 

the room-to-room search and quantify the detailed inspections of key features within the rooms. This 

scenario also totaled 100 points for a complete trial.

DRONE

SMALL

DUAL BUCKET

APPARATUS

CRUSHED VEHICLE SEARCH ROOM TO ROOM SEARCH

SMALL

DUAL BUCKET

APPARATUS

DRONE
Page 26 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Confined Scenario: Subway Terrorist Attack 

The subway terrorist attack scenario was in a simulated tunnel with multiple DC Metro railcars. A line of 

intact and upright railcars near the station entrance led to one railcar on its side almost blocking the tunnel. 

The scenario included several survivors inside and around the perimeter of the overturned railcar along 

with some suspicious packages that could have been secondary explosives. 

Figure 38) The long dark tunnel started with an intact set of railcars near at the simulated station entrance.

 

Figure 39) LEFT: We used Confined dual bucket alignments to designate 10 positions and orientations 

outside the subway car where victims were located along with suspicious packages. RIGHT: Two drones 

flew in tandem to help one another with side views of their situations as they negotiated the complex 

interior of the overturned railcar.
Page 27 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Figure 40) The quad screen video shows the first dual bucket alignment in the sideways entrance door. This 

was the first of 10 positions and orientations inside the subway car to measure how far in a drone could find 

victims and identify suspicious packages.

 

Figure 41) LEFT: The drones were assisted downrange by two four-legged robots. RIGHT: The environment 

was dark so thermal cameras were used extensively. This view is of the improvised pilot station on the 

subway tracks, set up near the overturned subway car rather than at the start of the tunnel reduce radio 

communications issues and flight times for the training scenario.

Legged Mobility and Search Tests

Newly developed standard terrain tests for robots with legged mobility are based on plastic crates. They 

are quick and easy to reconfigure into increasingly difficult terrains. For example, simply as a flat terrain 

they have a grate-like surface that some autonomous robots (and animals) don’t see so well. Various 
Page 28 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

terrains can be reconfigured including the Diagonal Hill (shown) or a Pyramid Hill with the highest point in 

the center of the room. Or every second crate can be flipped over in a checkered pattern to include 

negative obstacles or holes to avoid, forcing very specific footfall locations. These checkered terrains can be 

combined with the hill topologies for maximum difficulty. These crate-based terrain tests are also easy to 

purchase and fabricate all over the world. 

Note the white and black dual bucket alignment tasks all around the sides and top at different elevations 

and orientations. This is a quantitative way to measure and compare the search/inspect capabilities of 

robots while negotiating the terrains. The same search/inspect tasks can be placed over all the other 

standard terrains such as crossing ramps, sand and gravel crossover slopes, and stepfields, etc.

 

Figure 42) LEFT: The zig-zag crate terrain is easy to reconfigure for increasingly difficult challenges. CENTER: 

The diagonal hill with no holes. RIGHT: The checkered version of the terrain with holes either to avoid or 

deal with.

 

Figure 43) The Boston Dynamics Spot without its manipulator or extra sensors practicing the crate terrains.
Page 29 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 

Figure 44) LEFT: Embedded search/inspect tasks with dual bucket alignments that measure the difference 

between robots with and without manipulators that can position, orient, and zoom cameras while 

negotiating various terrains. The search/inspect tasks are exactly like the aerial Confined Test Lane tasks 

but require a closer 30 cm (12 in) standoff position to align with each pair of buckets simultaneously. RIGHT: 

Fetch tasks may be the next set of tests to standardize!

Sensor Test Lane

The Sensor Test Lane evaluates visual acuity, thermal acuity, hazmat label identification, motion detection, and 

latency while hovering at altitudes up to 90 m (300 ft). Also Survey Acuity for forward flying aircraft. It uses 

larger 120 liter (32 gallon) buckets with 45 cm (18 in) diameter inscribed rings to guide alignments. Panels 

with targets on white and black backgrounds point toward the hover position. Typical visual/thermal acuity 

targets are across the top line and operationally relevant objects are across the bottom line, including 

hazmat labels, partial license plates, gauges to read, weapons to identify, and thermal sources as examples. 

The back side of the target panels have QR codes of different sizes to capture in flyover video to measure 

the Survey Acuity of the system from various straight and level flight altitudes. The video can be postprocessed to automatically read as many incrementally small QR codes as possible for scoring. This can 

identify the smallest readable size across a large search area when multiple posters are laid out.
Page 30 11/9/21

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

 

Figure 45) LEFT: The Sensor Test Lane was set up on the centerline of an existing street. This makes it easy 

to ensure a straight line with measured distances between buckets and targets. The street also makes it 

easier to pre-program flight path using existing satellite imagery to fly straight and level over the centerline. 

RIGHT: The typical visual/thermal acuity targets on white and black backgrounds point toward the hover 

positions for the pilot to read. The back side of the A-frame panels have various size QR codes to read as a 

Survey Acuity test for forward flying aircraft.

-Adam

60 m (200 ft)

30 m (100 ft)